Nov 03 11:50:16.068061 ip-10-0-136-245 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 11:50:16.437887 ip-10-0-136-245 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:16.437887 ip-10-0-136-245 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 11:50:16.437887 ip-10-0-136-245 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:16.437887 ip-10-0-136-245 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443866    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443959    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443963    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443966    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443969    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443971    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443974    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.443964 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443977    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443980    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443983    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443986    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443988    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443991    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443994    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443996    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.443999    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444001    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444005    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444008    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444010    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444013    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444016    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444018    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444021    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444023    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444026    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444028    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.444248 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444031    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444034    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444036    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444039    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444041    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444044    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444046    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444048    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444051    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444054    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444056    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444059    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444061    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444064    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444067    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444070    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444072    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444075    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444077    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.444706 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444080    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444085    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444090    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444093    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444095    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444098    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444100    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444103    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444105    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444108    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444111    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444113    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444116    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444119    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444122    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444124    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444127    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444130    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.445171 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444132    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444135    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444137    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444140    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444142    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444145    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444147    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444150    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444153    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444156    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444158    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444161    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444164    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444166    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444169    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444172    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444174    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444177    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444179    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444183    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.445592 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444185    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444188    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444191    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444193    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444196    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444198    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444200    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444204    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444206    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444209    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444211    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444214    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444217    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444220    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444223    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444225    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444228    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444230    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444236    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444240    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.446099 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444242    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.444245    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445074    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445082    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445086    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445088    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445091    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445094    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445097    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445099    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445102    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445104    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445108    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445111    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445113    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445116    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445118    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445120    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445123    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.446563 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445125    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445128    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445131    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445133    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445137    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445139    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445142    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445144    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445147    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445149    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445153    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445156    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445158    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445161    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445163    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445166    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445168    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445171    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445174    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.447076 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445177    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445180    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445182    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445185    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445187    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445190    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445192    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445196    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445198    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445201    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445203    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445206    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445208    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445211    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445213    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445215    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445218    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445221    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445223    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.447533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445226    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445228    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445231    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445233    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445236    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445238    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445241    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445243    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445246    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445249    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445251    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445253    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445256    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445258    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445261    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445263    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445266    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445268    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445271    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445273    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.448049 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445276    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445279    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445281    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445284    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445286    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445289    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445294    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445296    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445299    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445302    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445305    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445307    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445310    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445312    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445315    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445317    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445320    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445322    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445325    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445327    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.448533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445329    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445334    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445337    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445341    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445345    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445348    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445351    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445354    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445357    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445359    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445362    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445891    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445896    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445900    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445902    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445905    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445908    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445911    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.449021 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445914    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445916    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445920    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445924    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445927    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445930    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445932    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445935    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445937    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445940    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445943    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445945    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445948    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445950    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445953    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445956    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445959    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445961    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445964    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.449851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445966    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445969    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445974    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445976    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445979    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445982    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445984    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445987    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445990    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445993    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445996    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.445998    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446001    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446004    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446006    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446009    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446011    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446014    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446016    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446018    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.450316 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446022    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446024    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446026    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446035    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446037    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446040    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446043    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446045    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446047    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446050    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446052    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446056    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446060    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446063    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446067    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446070    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446072    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446075    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446077    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.450851 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446080    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446082    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446085    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446089    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446092    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446094    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446097    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446100    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446102    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446105    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446108    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446111    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446114    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446116    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446119    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446122    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446124    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446127    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446129    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446132    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.451307 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446134    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446136    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446139    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446141    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446144    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446146    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446149    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446152    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446155    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446157    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446160    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446162    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446166    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446168    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446171    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446173    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446176    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446179    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446181    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446184    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.451867 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.446186    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447010    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447020    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447027    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447033    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447038    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447041    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447046    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447051    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447054    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447058    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447061    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447065    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447068    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447071    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447074    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447077    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447080    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447083    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447088    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447090    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447094    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447097    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447100    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 11:50:16.452331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447104    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447108    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447111    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447114    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447117    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447122    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447125    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447129    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447131    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447136    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447139    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447142    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447145    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447148    2582 flags.go:64] FLAG: --enable-server="true"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447151    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447156    2582 flags.go:64] FLAG: --event-burst="100"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447159    2582 flags.go:64] FLAG: --event-qps="50"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447162    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447165    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447168    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447172    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447175    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447178    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447181    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447184    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 11:50:16.453004 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447187    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447190    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447193    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447196    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447199    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447202    2582 flags.go:64] FLAG: --feature-gates=""
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447207    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447210    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447213    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447216    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447219    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447222    2582 flags.go:64] FLAG: --help="false"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447227    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447231    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447234    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447237    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447240    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447244    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447247    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447250    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447252    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447255    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447258    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 11:50:16.453621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447261    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447264    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447267    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447270    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447273    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447276    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447279    2582 flags.go:64] FLAG: --lock-file=""
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447282    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447285    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447288    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447294    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447297    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447300    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447303    2582 flags.go:64] FLAG: --logging-format="text"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447306    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447310    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447313    2582 flags.go:64] FLAG: --manifest-url=""
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447315    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447320    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447323    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447327    2582 flags.go:64] FLAG: --max-pods="110"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447331    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447334    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447338    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447341    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 11:50:16.454202 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447344    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447347    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447349    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447357    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447360    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447364    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447367    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447370    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447373    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447376    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447379    2582 flags.go:64] FLAG: --port="10250"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447382    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447385    2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01f9330680dc2b7c4"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447388    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447391    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447394    2582 flags.go:64] FLAG: --register-node="true"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447397    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447401    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447404    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447407    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447409    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447414    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447417    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447421    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447423    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447426    2582 flags.go:64] FLAG: --runonce="false"
Nov 03 11:50:16.454800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447429    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447433    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447436    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447439    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447442    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447447    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447450    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447453    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447456    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447459    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447462    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447465    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447468    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447471    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447475    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447481    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447484    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447487    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447490    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447493    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447496    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447499    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447502    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447505    2582 flags.go:64] FLAG: --v="2"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447510    2582 flags.go:64] FLAG: --version="false"
Nov 03 11:50:16.455461 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447514    2582 flags.go:64] FLAG: --vmodule=""
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447519    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.447522    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447664    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447668    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447670    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447673    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447678    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447682    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447687    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447690    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447693    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447697    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447701    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447705    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447708    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447712    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447715    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447718    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.456065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447721    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447724    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447728    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447730    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447733    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447736    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447738    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447741    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447743    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447761    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447763    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447766    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447769    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447771    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447774    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447776    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447779    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447782    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447784    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.456572 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447787    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447789    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447792    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447796    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447798    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447801    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447804    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447806    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447809    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447812    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447815    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447817    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447820    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447822    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447825    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447829    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447832    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447834    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447837    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447840    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.457062 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447843    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447845    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447848    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447850    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447853    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447856    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447858    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447861    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447864    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447866    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447869    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447871    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447874    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447876    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447878    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447882    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447885    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447888    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447890    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447893    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.457564 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447896    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447899    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447901    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447904    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447906    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447909    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447911    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447915    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447918    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447920    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447923    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447925    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447928    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447930    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447933    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447935    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447938    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447940    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447943    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.458048 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447945    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447948    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447950    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447953    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447955    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447958    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447960    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447963    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447966    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447969    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447972    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.447974    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.448001    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448042    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448046    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448050    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.458511 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448053    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448055    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448058    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448061    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448065    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448068    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448070    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448073    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448075    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448078    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448080    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448083    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448086    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448088    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448091    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448094    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448096    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448099    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448101    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.458974 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448104    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448108    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448112    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448115    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448118    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448120    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448124    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448127    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448130    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448134    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448138    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448141    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448144    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448147    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448150    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448153    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448155    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448159    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.459425 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448161    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448164    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448166    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448169    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448171    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448174    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448176    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448179    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448181    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448184    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448186    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448189    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448191    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448194    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448196    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448199    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448201    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448204    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448207    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448209    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.459938 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448213    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448215    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448218    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448220    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448223    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448225    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448228    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448230    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448233    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448236    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448238    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448242    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448245    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448247    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448249    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448252    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448254    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448257    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448260    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448262    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.460444 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448265    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448267    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448270    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448272    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448275    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448277    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448280    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448282    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448285    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448287    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448290    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448292    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448296    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448298    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448302    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448305    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448307    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448310    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448312    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448314    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.460923 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448317    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448320    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448322    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448326    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448329    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.448331    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.455538    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.455555    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.456827    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.456835    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456898    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456904    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456907    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456910    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456913    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456915    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456918    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456920    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456923    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.461405 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456926    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456928    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456932    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456934    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456937    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456939    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456942    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456944    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456947    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456949    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456952    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456954    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456957    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456960    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456963    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456966    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456969    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456971    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.461855 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456974    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456976    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456979    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456982    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456984    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456987    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456989    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456992    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456994    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456997    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.456999    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457002    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457004    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457007    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457010    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457012    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457015    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457017    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457020    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.462293 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457024    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457028    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457032    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457035    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457038    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457040    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457042    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457046    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457049    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457051    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457054    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457056    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457059    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457062    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457064    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457067    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457070    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457072    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457075    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457078    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.462791 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457081    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457083    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457086    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457088    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457091    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457093    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457096    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457098    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457101    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457103    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457106    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457108    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457110    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457113    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457116    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457118    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457121    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457123    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457126    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457128    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.463290 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457131    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457134    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457136    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457139    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457142    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457144    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457148    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457152    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457155    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457158    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457160    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457163    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457165    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457167    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457170    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457173    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457175    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457178    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457180    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.463764 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457183    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.457211    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457353    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457357    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457359    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457362    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457365    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457367    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457370    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457372    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457375    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457377    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457379    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457383    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457385    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457388    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.464243 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457390    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457393    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457396    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457398    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457401    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457404    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457406    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457408    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457411    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457413    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457416    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457418    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457421    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457424    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457426    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457428    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457431    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457433    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457436    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.464622 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457439    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457442    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457446    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457449    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457452    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457455    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457458    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457461    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457464    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457466    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457469    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457471    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457474    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457476    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457479    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457481    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457484    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457487    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457489    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.465423 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457492    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457494    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457497    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457500    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457503    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457507    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457510    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457513    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457515    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457518    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457521    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457523    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457526    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457528    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457531    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457533    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457536    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457539    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457541    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.466086 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457544    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457546    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457549    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457551    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457554    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457556    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457559    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457561    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457564    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457566    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457569    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457572    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457575    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457577    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457580    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457582    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457585    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457587    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457590    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457592    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.466557 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457595    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457597    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457600    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457603    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457605    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457607    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457610    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457612    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457615    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457617    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457620    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457622    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457625    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457627    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457629    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.457652    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:16.467064 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457684    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457688    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457691    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457693    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457696    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457699    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457701    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457705    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457707    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457710    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457712    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457715    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457718    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457720    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457723    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457725    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457728    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457730    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457733    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457735    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:16.467525 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457738    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457740    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457743    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457757    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457760    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457763    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457765    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457768    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457771    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457773    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457776    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457778    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457781    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457783    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457786    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457788    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457790    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457793    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457795    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457798    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:16.468115 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457809    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457812    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457814    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457817    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457819    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457822    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457824    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457827    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457829    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457832    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457838    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457840    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457843    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457845    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457848    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457850    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457854    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457856    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457859    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457862    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:16.468590 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457864    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457867    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457869    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457872    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457874    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457877    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457880    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457884    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457887    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457890    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457892    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457895    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457897    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457900    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457903    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457906    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457908    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457910    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457913    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:16.469065 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457915    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457918    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457920    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457924    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457927    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457929    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457932    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457936    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457938    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457942    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457946    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457950    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457953    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457956    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457959    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457962    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457964    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457967    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457970    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:16.469533 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457972    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457975    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457977    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457979    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457982    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457984    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457987    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:16.457990    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.458045    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.460239    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.461150    2582 server.go:1014] "Starting client certificate rotation"
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.461253    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.461288    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.463963    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:16.469984 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.468599    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:16.478081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.478064    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 03 11:50:16.481445 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.481431    2582 log.go:25] "Validated CRI v1 image API"
Nov 03 11:50:16.482278 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.482256    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 11:50:16.487571 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.487549    2582 fs.go:135] Filesystem UUIDs: map[4a323807-9fcb-4fcd-878d-a15fad11ee78:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c6ff839c-40e2-4202-9467-71074f89b07b:/dev/nvme0n1p3]
Nov 03 11:50:16.487621 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.487570    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 11:50:16.492655 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.492541    2582 manager.go:217] Machine: {Timestamp:2026-11-03 11:50:16.491158195 +0000 UTC m=+0.311363829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100746 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c5987f70ed9866df0a7d9936b42c SystemUUID:ec28c598-7f70-ed98-66df-0a7d9936b42c BootID:9479e698-c905-4b6e-bccf-9516e5c48247 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d7:40:3d:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d7:40:3d:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:07:ec:e8:21:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 11:50:16.492655 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.492657    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 11:50:16.492803 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.492790    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 11:50:16.495971 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.495948    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 11:50:16.496105 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.495972    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-245.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 11:50:16.496152 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.496107    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 11:50:16.496152 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.496116    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 11:50:16.496152 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.496151    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:16.496226 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.496170    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:16.497300 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.497289    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 11:50:16.501620 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.501607    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 11:50:16.501766 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.501741    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 11:50:16.502478 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.502468    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 11:50:16.502508 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.502483    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 11:50:16.502508 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.502494    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 11:50:16.502508 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.502506    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 03 11:50:16.502590 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.502525    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 11:50:16.502908 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.502896    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:16.505086 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.505062    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 11:50:16.511174 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.511153    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:16.511997 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.511982    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 11:50:16.512052 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512017    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 11:50:16.512176 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512166    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 11:50:16.512219 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512185    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 11:50:16.512219 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512192    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 11:50:16.512219 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512197    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 11:50:16.512219 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512203    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 11:50:16.512219 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512209    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 11:50:16.512786 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512777    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 11:50:16.512817 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512787    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 11:50:16.512817 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512794    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 11:50:16.512817 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512800    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 11:50:16.512817 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512808    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 11:50:16.512817 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.512816    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 11:50:16.514155 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.514145    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 11:50:16.514155 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.514155    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 11:50:16.517888 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.517875    2582 server.go:1263] "Started kubelet"
Nov 03 11:50:16.518015 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.517963    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 11:50:16.518015 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.518002    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 11:50:16.518151 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.518039    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 11:50:16.518676 ip-10-0-136-245 systemd[1]: Started Kubernetes Kubelet.
Nov 03 11:50:16.519468 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.519450    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 11:50:16.520543 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.520531    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 11:50:16.524046 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.524025    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 11:50:16.525344 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.525326    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 11:50:16.525447 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.525332    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:16.525447 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.525410    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:16.526014 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.525991    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 11:50:16.526150 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526132    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 11:50:16.526150 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526146    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 11:50:16.526291 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526204    2582 factory.go:55] Registering systemd factory
Nov 03 11:50:16.526291 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526222    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 03 11:50:16.526291 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526275    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 11:50:16.526291 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526283    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 11:50:16.526616 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526590    2582 factory.go:153] Registering CRI-O factory
Nov 03 11:50:16.526616 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526610    2582 factory.go:223] Registration of the crio container factory successfully
Nov 03 11:50:16.526721 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526665    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 11:50:16.526721 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526683    2582 factory.go:103] Registering Raw factory
Nov 03 11:50:16.526721 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.526696    2582 manager.go:1199] Started watching for new ooms in manager
Nov 03 11:50:16.527533 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.527519    2582 manager.go:319] Starting recovery of all containers
Nov 03 11:50:16.537079 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.537060    2582 manager.go:324] Recovery completed
Nov 03 11:50:16.544049 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.544029    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.546585 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.546563    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.546670 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.546599    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.546670 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.546610    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.547223 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.547207    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 03 11:50:16.547223 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.547219    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 11:50:16.547344 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.547236    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 11:50:16.547344 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.547260    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.548043 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.548025    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.548121 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.548053    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.548121 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.548064    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.549471 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.549460    2582 policy_none.go:50] "Start"
Nov 03 11:50:16.549513 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.549475    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 11:50:16.549513 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.549489    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 11:50:16.551490 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.551477    2582 policy_none.go:44] "Start"
Nov 03 11:50:16.591205 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.591003    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 11:50:16.603849 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.603831    2582 manager.go:341] "Starting Device Plugin manager"
Nov 03 11:50:16.603976 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.603871    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 11:50:16.603976 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.603886    2582 server.go:87] "Starting device plugin registration server"
Nov 03 11:50:16.604195 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.604169    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 11:50:16.604302 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.604196    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 11:50:16.604363 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.604299    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 11:50:16.604424 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.604411    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 11:50:16.604464 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.604439    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 11:50:16.605022 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.605003    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 11:50:16.605123 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.605069    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:16.652780 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.652733    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-245.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 11:50:16.652780 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.652743    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-245.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 11:50:16.654890 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.652723    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-245.us-west-2.compute.internal.18e486aa0cd06acb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-245.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-245.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:16.517847755 +0000 UTC m=+0.338053383,LastTimestamp:2026-11-03 11:50:16.517847755 +0000 UTC m=+0.338053383,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-245.us-west-2.compute.internal,}"
Nov 03 11:50:16.670371 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.670338    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 11:50:16.671724 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.671703    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 11:50:16.671829 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.671737    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 11:50:16.671829 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.671778    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 11:50:16.671829 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.671820    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 11:50:16.704425 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.704332    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.705469 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.705443    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.705593 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.705480    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.705593 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.705494    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.705593 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.705525    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.708037 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.707958    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-245.us-west-2.compute.internal.18e486aa0e86e3bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-245.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-245.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-245.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:16.546583487 +0000 UTC m=+0.366789117,LastTimestamp:2026-11-03 11:50:16.546583487 +0000 UTC m=+0.366789117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-245.us-west-2.compute.internal,}"
Nov 03 11:50:16.708159 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.708021    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 11:50:16.708159 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.708021    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-245.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 11:50:16.708159 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.708021    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 11:50:16.770181 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.770153    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 11:50:16.770297 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.770218    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d25j6"
Nov 03 11:50:16.772216 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.772197    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"]
Nov 03 11:50:16.772269 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.772260    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.774200 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.774184    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.774265 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.774212    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.774265 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.774225    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.776407 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.776395    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.776551 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.776539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.776587 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.776566    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.777285 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.777264    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.777285 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.777288    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.777413 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.777300    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.777413 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.777305    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.777413 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.777314    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.777413 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.777320    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.779499 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.779482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.779567 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.779510    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:16.780520 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.780505    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:16.780573 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.780532    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:16.780573 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.780567    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:16.805915 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.805894    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.806432 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.806331    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-245.us-west-2.compute.internal.18e486aa0e873413  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-245.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-245.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-245.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:16.546604051 +0000 UTC m=+0.366809680,LastTimestamp:2026-11-03 11:50:16.546604051 +0000 UTC m=+0.366809680,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-245.us-west-2.compute.internal,}"
Nov 03 11:50:16.808528 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.808510    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.827589 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.827563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.827589 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.827589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.827718 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.827607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cd06e439bdd68be11ac5140e2509e6a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5cd06e439bdd68be11ac5140e2509e6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.853915 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.853892    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:16.854388 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.854371    2582 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-136-245.us-west-2.compute.internal\" not found" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.928396 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.928356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.928396 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.928396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.928396 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.928414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cd06e439bdd68be11ac5140e2509e6a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5cd06e439bdd68be11ac5140e2509e6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.928631 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.928466    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.928631 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.928472    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.928631 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:16.928508    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cd06e439bdd68be11ac5140e2509e6a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5cd06e439bdd68be11ac5140e2509e6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:16.948529 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:16.948498    2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-245.us-west-2.compute.internal\" not found" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:17.054646 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.054568    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:17.055492 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.055475    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:17.055592 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.055505    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:17.055592 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.055515    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:17.055592 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.055544    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:17.106787 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.106764    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:17.106871 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.106790    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-245.us-west-2.compute.internal\": node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.110217 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.110197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:17.112358 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.112342    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:17.218842 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.218811    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.319330 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.319241    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.419507 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.419476    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.461761 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.461716    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 11:50:17.462204 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.461875    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 11:50:17.462204 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.461883    2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9e0eeeddb0cc409da088ac8135c41d1-80c4a411ee0df651.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-245.us-west-2.compute.internal.18e486aa0e873413\": read tcp 10.0.136.245:45310->13.184.82.147:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-245.us-west-2.compute.internal.18e486aa0e873413  default   5062 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-245.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-245.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-245.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:16 +0000 UTC,LastTimestamp:2026-11-03 11:50:16.774216632 +0000 UTC m=+0.594422262,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-245.us-west-2.compute.internal,}"
Nov 03 11:50:17.520319 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.520284    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.620576 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.620494    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.698119 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.698084    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:17.721505 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.721476    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.775615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.775539    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 11:45:17 +0000 UTC" deadline="2028-07-05 06:45:03.51481546 +0000 UTC"
Nov 03 11:50:17.775615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.775609    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14634h54m45.739211645s"
Nov 03 11:50:17.780713 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.780694    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:17.781744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.781726    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:17.781846 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.781773    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:17.781846 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.781783    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:17.821644 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.821611    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.922256 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:17.922179    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:17.996286 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:17.996256    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:18.026479 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.026454    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:18.085066 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.085036    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:18.086988 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.086903    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:18.144862 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.144810    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:18.221776 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.221727    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd06e439bdd68be11ac5140e2509e6a.slice/crio-1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473 WatchSource:0}: Error finding container 1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473: Status 404 returned error can't find the container with id 1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473
Nov 03 11:50:18.224612 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.224584    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcf5acf7ff1c4e9e6c3e7f2de3960f9.slice/crio-858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7 WatchSource:0}: Error finding container 858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7: Status 404 returned error can't find the container with id 858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7
Nov 03 11:50:18.229741 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.229725    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:50:18.454908 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.454873    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:18.506002 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.505910    2582 apiserver.go:52] "Watching apiserver"
Nov 03 11:50:18.527757 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.527719    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:18.611658 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.611612    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 11:50:18.612012 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.611987    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z","openshift-dns/node-resolver-ssfqd","openshift-multus/network-metrics-daemon-s79nv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal","openshift-multus/multus-2f4bq","openshift-network-operator/iptables-alerter-hv5fv","openshift-ovn-kubernetes/ovnkube-node-sqqxm","openshift-cluster-node-tuning-operator/tuned-j62p5","openshift-image-registry/node-ca-vkpbq","openshift-multus/multus-additional-cni-plugins-rq7tf","openshift-network-diagnostics/network-check-target-pgfhl","kube-system/global-pull-secret-syncer-lpbww","kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal","kube-system/konnectivity-agent-wh554"]
Nov 03 11:50:18.616052 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.616026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.618342 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.618312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.620629 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.620608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:18.620708 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.620687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:18.622930 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.622915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.625490 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.625474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.628342 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.628325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:18.628447 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.628388    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:18.630697 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.630678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.633093 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.633077    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.635494 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.635477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.637840 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.637813    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.638161 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.638254 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.638254 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-var-lib-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638254 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638254 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-netns\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638451 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-etc-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638451 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638451 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-env-overrides\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638565 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45488046-69c7-4931-b7b7-6efe7be609df-ovn-node-metrics-cert\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638565 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56tkk\" (UniqueName: \"kubernetes.io/projected/45488046-69c7-4931-b7b7-6efe7be609df-kube-api-access-56tkk\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638565 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-system-cni-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.638709 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-log-socket\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638709 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.638709 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-kubelet\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638709 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-systemd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638709 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b075924a-500c-4f50-ab6c-dbb51839181c-hosts-file\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z49h\" (UniqueName: \"kubernetes.io/projected/dd63d182-450e-45aa-a997-4c98efe630dc-kube-api-access-4z49h\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-node-log\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075924a-500c-4f50-ab6c-dbb51839181c-tmp-dir\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-konnectivity-ca\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-systemd-units\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-netd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-script-lib\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.638964 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.638974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-agent-certs\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-slash\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-bin\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-cnibin\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpccs\" (UniqueName: \"kubernetes.io/projected/b075924a-500c-4f50-ab6c-dbb51839181c-kube-api-access-gpccs\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57x2n\" (UniqueName: \"kubernetes.io/projected/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-kube-api-access-57x2n\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-ovn\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.639365 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.639290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-config\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.641017 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.640998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.641100 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.641078    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:18.643713 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.643691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.670797 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.670752    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.670797 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.670752    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 11:50:18.670953 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.670782    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.670953 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.670847    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 11:50:18.670953 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.670873    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jp68j\""
Nov 03 11:50:18.670953 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.670909    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 11:50:18.671649 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.671633    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.671721 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.671710    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xf9nj\""
Nov 03 11:50:18.671721 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.671714    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.676245 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.676226    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 11:50:18.676353 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.676309    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-swh4m\""
Nov 03 11:50:18.676421 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.676412    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 11:50:18.676863 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.676818    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5bcf5acf7ff1c4e9e6c3e7f2de3960f9","Type":"ContainerStarted","Data":"858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7"}
Nov 03 11:50:18.678298 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.678273    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5cd06e439bdd68be11ac5140e2509e6a","Type":"ContainerStarted","Data":"1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473"}
Nov 03 11:50:18.683011 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.682992    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 11:50:18.683125 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.683008    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.683125 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.683024    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 11:50:18.683125 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.683085    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.683125 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.683097    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 11:50:18.683298 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.683178    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jp4k\""
Nov 03 11:50:18.683298 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.683254    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 11:50:18.684289 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.684275    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 11:50:18.726999 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.726975    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 11:50:18.739491 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-sys\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.739615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-tuned\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.739615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b075924a-500c-4f50-ab6c-dbb51839181c-hosts-file\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.739615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z49h\" (UniqueName: \"kubernetes.io/projected/dd63d182-450e-45aa-a997-4c98efe630dc-kube-api-access-4z49h\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.739615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-node-log\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.739615 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-socket-dir-parent\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-k8s-cni-cncf-io\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075924a-500c-4f50-ab6c-dbb51839181c-tmp-dir\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739654    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b075924a-500c-4f50-ab6c-dbb51839181c-hosts-file\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-konnectivity-ca\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-netd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-script-lib\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.739910 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739877    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-node-log\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-netd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.739969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-serviceca\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-bin\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-etc-kubernetes\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-host\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075924a-500c-4f50-ab6c-dbb51839181c-tmp-dir\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-kubernetes\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740170    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-bin\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-run\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.740264 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-cnibin\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpccs\" (UniqueName: \"kubernetes.io/projected/b075924a-500c-4f50-ab6c-dbb51839181c-kube-api-access-gpccs\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57x2n\" (UniqueName: \"kubernetes.io/projected/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-kube-api-access-57x2n\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740411    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-cnibin\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740439    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-konnectivity-ca\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-cnibin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntspk\" (UniqueName: \"kubernetes.io/projected/e5e9caab-b867-45ec-b81c-b74d1821302d-kube-api-access-ntspk\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-device-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.740819 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-modprobe-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-conf\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-netns\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-etc-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-kubelet\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-log-socket\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740970    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-netns\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-etc-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-system-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.740995    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-log-socket\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcjz\" (UniqueName: \"kubernetes.io/projected/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kube-api-access-7rcjz\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-kubelet\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-kubelet\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-systemd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-daemon-config\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.741262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-host\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741228    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-systemd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-multus\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741314    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741323    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-multus-certs\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741369    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-var-lib-kubelet\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-systemd-units\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741417    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-script-lib\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741444    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-systemd-units\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-cni-binary-copy\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-agent-certs\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.741553    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-slash\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742081 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-netns\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741608    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.741629    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:50:19.241607836 +0000 UTC m=+3.061813472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741651    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-slash\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-lib-modules\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7lm\" (UniqueName: \"kubernetes.io/projected/fdbf6c89-33f3-48a2-aa96-16a26e336e34-kube-api-access-hf7lm\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-ovn\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-config\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-conf-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741833    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-iptables-alerter-script\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741881    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-ovn\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-tmp\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-var-lib-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.742805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741930    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-hostroot\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741979    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-var-lib-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.741980    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-env-overrides\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45488046-69c7-4931-b7b7-6efe7be609df-ovn-node-metrics-cert\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742055    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56tkk\" (UniqueName: \"kubernetes.io/projected/45488046-69c7-4931-b7b7-6efe7be609df-kube-api-access-56tkk\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-bin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5k9m\" (UniqueName: \"kubernetes.io/projected/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-kube-api-access-t5k9m\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-system-cni-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742253    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-system-cni-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742265    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-config\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.743805 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-host-slash\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv992\" (UniqueName: \"kubernetes.io/projected/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-kube-api-access-mv992\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-kubelet-config\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-env-overrides\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-dbus\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysconfig\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-systemd\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.744677 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.742568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.745659 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.745640    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45488046-69c7-4931-b7b7-6efe7be609df-ovn-node-metrics-cert\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.745747 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.745731    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-agent-certs\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.774095 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.774028    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 11:50:18.774095 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.774078    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4w8ct"
Nov 03 11:50:18.779177 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779160    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 11:50:18.779376 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779359    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 11:50:18.779478 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779461    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.779524 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779496    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6gwtx\""
Nov 03 11:50:18.779524 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779508    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.779818 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779796    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.779900 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779843    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.779974 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.779899    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.780123 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780081    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 11:50:18.780123 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780100    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:18.780123 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780104    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-65pss\""
Nov 03 11:50:18.780329 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780139    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vn7bf\""
Nov 03 11:50:18.780329 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780177    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.780329 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780294    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:18.780556 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780535    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxdwl\""
Nov 03 11:50:18.780641 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.780629    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g5g7l\""
Nov 03 11:50:18.799181 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.799156    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:18.799181 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.799179    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:18.799344 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.799190    2582 projected.go:196] Error preparing data for projected volume kube-api-access-tknkk for pod openshift-network-diagnostics/network-check-target-pgfhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:18.799344 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.799263    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk podName:62d43ee3-52a9-4c5f-80f1-81b795f04ea6 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:19.299242859 +0000 UTC m=+3.119448496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tknkk" (UniqueName: "kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk") pod "network-check-target-pgfhl" (UID: "62d43ee3-52a9-4c5f-80f1-81b795f04ea6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:18.801966 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.801940    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56tkk\" (UniqueName: \"kubernetes.io/projected/45488046-69c7-4931-b7b7-6efe7be609df-kube-api-access-56tkk\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.802282 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.802265    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z49h\" (UniqueName: \"kubernetes.io/projected/dd63d182-450e-45aa-a997-4c98efe630dc-kube-api-access-4z49h\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.803001 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.802978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57x2n\" (UniqueName: \"kubernetes.io/projected/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-kube-api-access-57x2n\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:18.803099 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.803075    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpccs\" (UniqueName: \"kubernetes.io/projected/b075924a-500c-4f50-ab6c-dbb51839181c-kube-api-access-gpccs\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.843867 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.843838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-system-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.843867 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.843873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.844084 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.843892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcjz\" (UniqueName: \"kubernetes.io/projected/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kube-api-access-7rcjz\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.844084 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.843909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-daemon-config\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844084 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.843972    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.844084 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.843986    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-system-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844283 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-host\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844283 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-multus\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844283 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-multus-certs\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844417 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844297    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-host\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844417 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844330    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-multus\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844417 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-var-lib-kubelet\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844525 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-multus-certs\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844525 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-cni-binary-copy\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844525 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844440    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-var-lib-kubelet\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844525 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-netns\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844525 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-lib-modules\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7lm\" (UniqueName: \"kubernetes.io/projected/fdbf6c89-33f3-48a2-aa96-16a26e336e34-kube-api-access-hf7lm\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844579    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-netns\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-conf-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844601    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-iptables-alerter-script\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844662    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-conf-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-tmp\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844690    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-daemon-config\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-hostroot\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.844744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844741    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-lib-modules\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-bin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844800    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844745    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-hostroot\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844852    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-bin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k9m\" (UniqueName: \"kubernetes.io/projected/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-kube-api-access-t5k9m\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844879    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-host-slash\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv992\" (UniqueName: \"kubernetes.io/projected/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-kube-api-access-mv992\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844963    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-host-slash\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.844965    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-kubelet-config\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845057    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-cni-binary-copy\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-dbus\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-kubelet-config\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysconfig\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.845393 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-systemd\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-dbus\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-sys\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-tuned\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845252    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-systemd\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-socket-dir-parent\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845303    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.845326    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysconfig\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-k8s-cni-cncf-io\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845361    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-sys\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845363    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-socket-dir-parent\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:18.845389    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret podName:606502b8-aa74-4661-94fe-43d62bfb8d40 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:19.345368233 +0000 UTC m=+3.165573855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret") pod "global-pull-secret-syncer-lpbww" (UID: "606502b8-aa74-4661-94fe-43d62bfb8d40") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845360    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-k8s-cni-cncf-io\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.846115 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-serviceca\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-etc-kubernetes\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-host\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-etc-kubernetes\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-kubernetes\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-run\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845596    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-iptables-alerter-script\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845602    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-host\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845668    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-run\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845671    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-kubernetes\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-cnibin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntspk\" (UniqueName: \"kubernetes.io/projected/e5e9caab-b867-45ec-b81c-b74d1821302d-kube-api-access-ntspk\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-device-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-modprobe-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-conf\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845839    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-device-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-kubelet\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.846594 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845885    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-kubelet\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.847135 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845841    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-cnibin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.847135 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.845994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-modprobe-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.847135 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.846088    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-conf\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.847135 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.846190    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-serviceca\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.848188 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.848164    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-tmp\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.848285 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.848208    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-tuned\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.890880 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.890851    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:18.903903 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.903877    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv992\" (UniqueName: \"kubernetes.io/projected/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-kube-api-access-mv992\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.903903 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.903901    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntspk\" (UniqueName: \"kubernetes.io/projected/e5e9caab-b867-45ec-b81c-b74d1821302d-kube-api-access-ntspk\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.904111 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.903938    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7lm\" (UniqueName: \"kubernetes.io/projected/fdbf6c89-33f3-48a2-aa96-16a26e336e34-kube-api-access-hf7lm\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.904268 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.904247    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcjz\" (UniqueName: \"kubernetes.io/projected/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kube-api-access-7rcjz\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.904324 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.904256    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k9m\" (UniqueName: \"kubernetes.io/projected/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-kube-api-access-t5k9m\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.939057 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.939022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:18.943043 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.943011    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:18.948996 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.948967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:18.949464 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.949443    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb075924a_500c_4f50_ab6c_dbb51839181c.slice/crio-50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1 WatchSource:0}: Error finding container 50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1: Status 404 returned error can't find the container with id 50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1
Nov 03 11:50:18.951708 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.951681    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ead3a7f_804d_48a7_9dae_ec5df206cbbe.slice/crio-53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa WatchSource:0}: Error finding container 53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa: Status 404 returned error can't find the container with id 53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa
Nov 03 11:50:18.958970 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.958947    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd63d182_450e_45aa_a997_4c98efe630dc.slice/crio-0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c WatchSource:0}: Error finding container 0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c: Status 404 returned error can't find the container with id 0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c
Nov 03 11:50:18.959500 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.959482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:18.962157 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.962139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:18.969763 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.969743    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e9caab_b867_45ec_b81c_b74d1821302d.slice/crio-d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db WatchSource:0}: Error finding container d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db: Status 404 returned error can't find the container with id d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db
Nov 03 11:50:18.970913 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.970895    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:18.971838 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.971756    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67983f1_d777_4fde_a01b_24d8a2a4f3c5.slice/crio-d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca WatchSource:0}: Error finding container d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca: Status 404 returned error can't find the container with id d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca
Nov 03 11:50:18.977934 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.977920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:18.980439 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.980418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:18.980817 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.980791    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45488046_69c7_4931_b7b7_6efe7be609df.slice/crio-5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f WatchSource:0}: Error finding container 5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f: Status 404 returned error can't find the container with id 5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f
Nov 03 11:50:18.983187 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:18.983173    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:18.990090 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.990062    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bee7ca_d883_4a3b_94c7_9ad031cad22f.slice/crio-a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4 WatchSource:0}: Error finding container a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4: Status 404 returned error can't find the container with id a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4
Nov 03 11:50:18.990990 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.990969    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b8775a_e0e0_4cf6_9be5_9df4891d6ccc.slice/crio-1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4 WatchSource:0}: Error finding container 1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4: Status 404 returned error can't find the container with id 1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4
Nov 03 11:50:18.993308 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:18.993290    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbf6c89_33f3_48a2_aa96_16a26e336e34.slice/crio-21acab038a6188e07c6566acbddee8f6f91f42475918e6b227f8e5c97a024b46 WatchSource:0}: Error finding container 21acab038a6188e07c6566acbddee8f6f91f42475918e6b227f8e5c97a024b46: Status 404 returned error can't find the container with id 21acab038a6188e07c6566acbddee8f6f91f42475918e6b227f8e5c97a024b46
Nov 03 11:50:19.249632 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.249553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:19.249853 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.249704    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:19.249853 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.249798    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:50:20.24977481 +0000 UTC m=+4.069980443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:19.350722 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.350675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:19.350935 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.350736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:19.351018 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.350965    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:19.351018 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.350987    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:19.351018 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.350997    2582 projected.go:196] Error preparing data for projected volume kube-api-access-tknkk for pod openshift-network-diagnostics/network-check-target-pgfhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:19.351176 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.351050    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk podName:62d43ee3-52a9-4c5f-80f1-81b795f04ea6 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:20.351032265 +0000 UTC m=+4.171237884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tknkk" (UniqueName: "kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk") pod "network-check-target-pgfhl" (UID: "62d43ee3-52a9-4c5f-80f1-81b795f04ea6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:19.351176 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.351129    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:19.351176 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:19.351166    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret podName:606502b8-aa74-4661-94fe-43d62bfb8d40 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:20.351154665 +0000 UTC m=+4.171360305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret") pod "global-pull-secret-syncer-lpbww" (UID: "606502b8-aa74-4661-94fe-43d62bfb8d40") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:19.682030 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.681934    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j62p5" event={"ID":"fdbf6c89-33f3-48a2-aa96-16a26e336e34","Type":"ContainerStarted","Data":"21acab038a6188e07c6566acbddee8f6f91f42475918e6b227f8e5c97a024b46"}
Nov 03 11:50:19.683420 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.683389    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hv5fv" event={"ID":"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc","Type":"ContainerStarted","Data":"1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4"}
Nov 03 11:50:19.684995 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.684966    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f"}
Nov 03 11:50:19.686799 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.686774    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2f4bq" event={"ID":"e5e9caab-b867-45ec-b81c-b74d1821302d","Type":"ContainerStarted","Data":"d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db"}
Nov 03 11:50:19.688169 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.688147    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wh554" event={"ID":"2ead3a7f-804d-48a7-9dae-ec5df206cbbe","Type":"ContainerStarted","Data":"53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa"}
Nov 03 11:50:19.689705 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.689676    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z" event={"ID":"a9bee7ca-d883-4a3b-94c7-9ad031cad22f","Type":"ContainerStarted","Data":"a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4"}
Nov 03 11:50:19.691026 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.691003    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vkpbq" event={"ID":"c67983f1-d777-4fde-a01b-24d8a2a4f3c5","Type":"ContainerStarted","Data":"d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca"}
Nov 03 11:50:19.692263 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.692235    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerStarted","Data":"0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c"}
Nov 03 11:50:19.693788 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.693738    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ssfqd" event={"ID":"b075924a-500c-4f50-ab6c-dbb51839181c","Type":"ContainerStarted","Data":"50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1"}
Nov 03 11:50:19.779292 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.779224    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:19 +0000 UTC" deadline="2028-07-03 06:27:07.352521967 +0000 UTC"
Nov 03 11:50:19.779292 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:19.779287    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14586h36m47.573240652s"
Nov 03 11:50:20.258510 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.258476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:20.258697 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.258660    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:20.258779 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.258725    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:50:22.258710442 +0000 UTC m=+6.078916061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:20.359672 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.359617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:20.360022 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.359711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:20.363024 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.362994    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:20.363150 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.363105    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:20.363150 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.363109    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret podName:606502b8-aa74-4661-94fe-43d62bfb8d40 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:22.363069676 +0000 UTC m=+6.183275313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret") pod "global-pull-secret-syncer-lpbww" (UID: "606502b8-aa74-4661-94fe-43d62bfb8d40") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:20.363150 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.363135    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:20.363314 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.363153    2582 projected.go:196] Error preparing data for projected volume kube-api-access-tknkk for pod openshift-network-diagnostics/network-check-target-pgfhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:20.363314 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.363220    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk podName:62d43ee3-52a9-4c5f-80f1-81b795f04ea6 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:22.363202136 +0000 UTC m=+6.183407758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tknkk" (UniqueName: "kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk") pod "network-check-target-pgfhl" (UID: "62d43ee3-52a9-4c5f-80f1-81b795f04ea6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:20.675626 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.675521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:20.675790 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.675676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:20.675891 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.675797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:20.675959 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.675894    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:20.676015 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.675971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:20.676068 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:20.676049    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:20.780210 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.780097    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:19 +0000 UTC" deadline="2028-06-14 01:23:58.261718541 +0000 UTC"
Nov 03 11:50:20.780716 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:20.780210    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h33m37.48151513s"
Nov 03 11:50:22.280818 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.280428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:22.280818 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.280650    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:22.280818 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.280721    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:50:26.280700514 +0000 UTC m=+10.100906338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.380939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.381005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.381176    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.381195    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.381211    2582 projected.go:196] Error preparing data for projected volume kube-api-access-tknkk for pod openshift-network-diagnostics/network-check-target-pgfhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.381270    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk podName:62d43ee3-52a9-4c5f-80f1-81b795f04ea6 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:26.381251228 +0000 UTC m=+10.201456851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tknkk" (UniqueName: "kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk") pod "network-check-target-pgfhl" (UID: "62d43ee3-52a9-4c5f-80f1-81b795f04ea6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.381359    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:22.381504 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.381437    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret podName:606502b8-aa74-4661-94fe-43d62bfb8d40 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:26.381423937 +0000 UTC m=+10.201629561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret") pod "global-pull-secret-syncer-lpbww" (UID: "606502b8-aa74-4661-94fe-43d62bfb8d40") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:22.675723 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.675454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:22.675982 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.675917    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:22.676064 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.676034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:22.676161 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.676136    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:22.676221 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.676205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:22.676297 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:22.676276    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:22.720318 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.720199    2582 generic.go:356] "Generic (PLEG): container finished" podID="5bcf5acf7ff1c4e9e6c3e7f2de3960f9" containerID="927fe6d6f023e6b06c3c0e6dd2b0d3cb08257c9761e99be8f4bf38b08c0fe72b" exitCode=0
Nov 03 11:50:22.720318 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:22.720254    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5bcf5acf7ff1c4e9e6c3e7f2de3960f9","Type":"ContainerDied","Data":"927fe6d6f023e6b06c3c0e6dd2b0d3cb08257c9761e99be8f4bf38b08c0fe72b"}
Nov 03 11:50:23.726708 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:23.726668    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5bcf5acf7ff1c4e9e6c3e7f2de3960f9","Type":"ContainerStarted","Data":"4cf3fa5e806f8516ce87c32a4c2a10d543472e423407aa8b352a8d78926fac65"}
Nov 03 11:50:23.733089 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:23.733041    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="f9f3e7134df583d375db2dde2bf8a824f534167f81b1d0b88be59ba338995131" exitCode=0
Nov 03 11:50:23.733310 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:23.733178    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerDied","Data":"f9f3e7134df583d375db2dde2bf8a824f534167f81b1d0b88be59ba338995131"}
Nov 03 11:50:23.739598 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:23.739461    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5cd06e439bdd68be11ac5140e2509e6a","Type":"ContainerStarted","Data":"a599be0a5a8491b2c12d062cf747779292f3c49a3ccdc181ee0b2316943931a9"}
Nov 03 11:50:23.844330 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:23.844248    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" podStartSLOduration=5.844225371 podStartE2EDuration="5.844225371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:23.843602872 +0000 UTC m=+7.663808515" watchObservedRunningTime="2026-11-03 11:50:23.844225371 +0000 UTC m=+7.664431014"
Nov 03 11:50:24.675861 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:24.675176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:24.675861 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:24.675317    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:24.675861 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:24.675447    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:24.675861 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:24.675544    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:24.675861 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:24.675606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:24.675861 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:24.675674    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:24.745913 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:24.745476    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wh554" event={"ID":"2ead3a7f-804d-48a7-9dae-ec5df206cbbe","Type":"ContainerStarted","Data":"1ea96b5213e2b8b7be15f7db2458bc50104919c2d713568ad3c1a3f64fc5a698"}
Nov 03 11:50:24.861954 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:24.861886    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal" podStartSLOduration=6.861861605 podStartE2EDuration="6.861861605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:24.092623292 +0000 UTC m=+7.912828934" watchObservedRunningTime="2026-11-03 11:50:24.861861605 +0000 UTC m=+8.682067247"
Nov 03 11:50:25.750339 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:25.750299    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vkpbq" event={"ID":"c67983f1-d777-4fde-a01b-24d8a2a4f3c5","Type":"ContainerStarted","Data":"1e65d8a57571e5ef93582f797e6bf1403c73a6511dc0520add3b7dfbb83e1ae1"}
Nov 03 11:50:25.868260 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:25.866972    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wh554" podStartSLOduration=3.953711035 podStartE2EDuration="8.866951023s" totalImagesPullingTime="4.913239988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:24.862049684 +0000 UTC m=+8.682255325" watchObservedRunningTime="2026-11-03 11:50:25.866951023 +0000 UTC m=+9.687156667"
Nov 03 11:50:26.317310 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.317223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:26.317550 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.317349    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:26.317550 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.317408    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:50:34.317389752 +0000 UTC m=+18.137595385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:26.418052 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.418006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:26.418212 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.418074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:26.418291 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.418242    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:26.418291 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.418262    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:26.418291 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.418278    2582 projected.go:196] Error preparing data for projected volume kube-api-access-tknkk for pod openshift-network-diagnostics/network-check-target-pgfhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:26.418458 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.418341    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk podName:62d43ee3-52a9-4c5f-80f1-81b795f04ea6 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:34.418321554 +0000 UTC m=+18.238527200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tknkk" (UniqueName: "kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk") pod "network-check-target-pgfhl" (UID: "62d43ee3-52a9-4c5f-80f1-81b795f04ea6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:26.418458 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.418425    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:26.418583 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.418464    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret podName:606502b8-aa74-4661-94fe-43d62bfb8d40 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:34.418451504 +0000 UTC m=+18.238657137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret") pod "global-pull-secret-syncer-lpbww" (UID: "606502b8-aa74-4661-94fe-43d62bfb8d40") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:26.684265 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.684195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:26.684401 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.684326    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:26.684401 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.684385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:26.684479 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.684449    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:26.684900 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.684877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:26.687013 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:26.685034    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:26.757223 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:26.757183    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z" event={"ID":"a9bee7ca-d883-4a3b-94c7-9ad031cad22f","Type":"ContainerStarted","Data":"defd0c6e66ed8211c380490d309eea34f2551ede4fc0013d72d8d9db3f2f48ac"}
Nov 03 11:50:27.765562 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:27.765522    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2f4bq" event={"ID":"e5e9caab-b867-45ec-b81c-b74d1821302d","Type":"ContainerStarted","Data":"f61715b68a2244e0724983146b3391d0c353569fda3a5c9932a81376e81b673b"}
Nov 03 11:50:28.196091 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.196057    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 11:50:28.545252 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.545157    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:28.545971 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.545950    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:28.613951 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.613843    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T11:50:28.196081743Z","UUID":"015939a6-7cf5-48eb-ac37-2da63698fe28","Handler":null,"Name":"","Endpoint":""}
Nov 03 11:50:28.615783 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.615739    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 11:50:28.615928 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.615791    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 11:50:28.662683 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.662624    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vkpbq" podStartSLOduration=4.609088399 podStartE2EDuration="10.662602971s" totalImagesPullingTime="6.053514572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:25.868227459 +0000 UTC m=+9.688433101" watchObservedRunningTime="2026-11-03 11:50:28.662602971 +0000 UTC m=+12.482808671"
Nov 03 11:50:28.674110 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.674078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:28.674275 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:28.674214    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:28.674633 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.674597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:28.674728 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:28.674708    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:28.674815 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.674759    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:28.674871 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:28.674844    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:28.769940 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.769896    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ssfqd" event={"ID":"b075924a-500c-4f50-ab6c-dbb51839181c","Type":"ContainerStarted","Data":"ffe836173b4c27699c9bf8d3db5f44b443237589164b285a37a0c024845078bf"}
Nov 03 11:50:28.773728 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.773697    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z" event={"ID":"a9bee7ca-d883-4a3b-94c7-9ad031cad22f","Type":"ContainerStarted","Data":"d1b82e04b038ee5f65a504d8f8fd9ef289c9cd12f3ef6f5dd68ce8b671d8d11f"}
Nov 03 11:50:28.773885 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.773762    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:28.774717 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:28.774694    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:29.003539 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:29.003427    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ssfqd" podStartSLOduration=3.05060469 podStartE2EDuration="12.003411514s" totalImagesPullingTime="8.952806824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:28.887162851 +0000 UTC m=+12.707368495" watchObservedRunningTime="2026-11-03 11:50:29.003411514 +0000 UTC m=+12.823617167"
Nov 03 11:50:29.778650 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:29.778408    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hv5fv" event={"ID":"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc","Type":"ContainerStarted","Data":"3d3b62cd807ffdfebdc55874faafd08e9648fbc9ccfffba3077fb97074d35cb5"}
Nov 03 11:50:30.672883 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:30.672804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:30.672883 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:30.672801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:30.673086 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:30.673028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:30.673086 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:30.673036    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:30.673180 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:30.673133    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:30.673235 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:30.673216    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:30.782987 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:30.782947    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z" event={"ID":"a9bee7ca-d883-4a3b-94c7-9ad031cad22f","Type":"ContainerStarted","Data":"7b50eb5439e6b6b29150b48b126123a40bc731101abdc6d073ee3decf87dcce6"}
Nov 03 11:50:30.785141 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:30.785111    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j62p5" event={"ID":"fdbf6c89-33f3-48a2-aa96-16a26e336e34","Type":"ContainerStarted","Data":"c5b5835990d2c6ba88b4caea16084b18199a6a704d47f0b9f1f5dee825e28221"}
Nov 03 11:50:30.940322 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:30.940226    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2f4bq" podStartSLOduration=4.261607214 podStartE2EDuration="12.940206309s" totalImagesPullingTime="8.678599095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:29.123085449 +0000 UTC m=+12.943291073" watchObservedRunningTime="2026-11-03 11:50:30.940206309 +0000 UTC m=+14.760411948"
Nov 03 11:50:31.048968 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:31.048910    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z" podStartSLOduration=2.005636852 podStartE2EDuration="13.048895439s" totalImagesPullingTime="11.043258587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:30.941108713 +0000 UTC m=+14.761314353" watchObservedRunningTime="2026-11-03 11:50:31.048895439 +0000 UTC m=+14.869101077"
Nov 03 11:50:31.157270 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:31.157218    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hv5fv" podStartSLOduration=3.935119401 podStartE2EDuration="13.157205425s" totalImagesPullingTime="9.222086024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:31.048409731 +0000 UTC m=+14.868615371" watchObservedRunningTime="2026-11-03 11:50:31.157205425 +0000 UTC m=+14.977411063"
Nov 03 11:50:31.157497 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:31.157464    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j62p5" podStartSLOduration=1.7863149539999998 podStartE2EDuration="13.157455605s" totalImagesPullingTime="11.371140651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:31.157010201 +0000 UTC m=+14.977215839" watchObservedRunningTime="2026-11-03 11:50:31.157455605 +0000 UTC m=+14.977661260"
Nov 03 11:50:31.789886 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:31.789806    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="d85c0b7352c2700d65640f7479fbfcaeeb5a8606a5a3ebed25289dd5de730e3f" exitCode=0
Nov 03 11:50:31.789886 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:31.789873    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerDied","Data":"d85c0b7352c2700d65640f7479fbfcaeeb5a8606a5a3ebed25289dd5de730e3f"}
Nov 03 11:50:32.672726 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:32.672692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:32.672726 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:32.672710    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:32.672959 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:32.672847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:32.672959 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:32.672848    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:32.673060 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:32.672984    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:32.673130 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:32.673103    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:33.796581 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:33.796354    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="8597a652670ed492b99ef874be7c269e1b9ad5619a50707debd790a19a951360" exitCode=0
Nov 03 11:50:33.796581 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:33.796443    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerDied","Data":"8597a652670ed492b99ef874be7c269e1b9ad5619a50707debd790a19a951360"}
Nov 03 11:50:34.379706 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.379678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:34.379851 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.379802    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:34.379887 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.379856    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.379843788 +0000 UTC m=+34.200049417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:34.479985 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.479953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:34.480064 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.480000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:34.480114 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.480097    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:34.480114 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.480110    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:34.480185 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.480125    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:34.480185 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.480137    2582 projected.go:196] Error preparing data for projected volume kube-api-access-tknkk for pod openshift-network-diagnostics/network-check-target-pgfhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:34.480185 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.480168    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret podName:606502b8-aa74-4661-94fe-43d62bfb8d40 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.480152893 +0000 UTC m=+34.300358531 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret") pod "global-pull-secret-syncer-lpbww" (UID: "606502b8-aa74-4661-94fe-43d62bfb8d40") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:34.480185 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.480184    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk podName:62d43ee3-52a9-4c5f-80f1-81b795f04ea6 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.480177013 +0000 UTC m=+34.300382630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tknkk" (UniqueName: "kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk") pod "network-check-target-pgfhl" (UID: "62d43ee3-52a9-4c5f-80f1-81b795f04ea6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:34.672343 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.672303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:34.672343 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.672333    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:34.672578 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.672303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:34.672578 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.672414    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:34.672578 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.672476    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:34.672578 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:34.672533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:34.802703 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.802604    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"885285130fe36d62b8843229b134efc3418eef1d4226fb4858e1ef21920ab415"}
Nov 03 11:50:34.802703 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.802689    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"06620cfcf91fc5c6c864df95ecd5c0a11873363c0371d8e1bd43e69687a0a9c6"}
Nov 03 11:50:34.803186 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.802720    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"339e2154ca0056bb422b413f06bbec7d3161cbdbda1432bb61aea21a94e4910d"}
Nov 03 11:50:34.803186 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.802778    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"7cbd5dd64563b077c004ec6799bec8a8c12f9b427491fd9344397d9a51d1fdce"}
Nov 03 11:50:34.803186 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.802812    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"301500519b3892e1ce7ff2917b00030bd2e19c5b6cf17ee127950004d023933c"}
Nov 03 11:50:34.803186 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:34.802841    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"9d5a6fb6ed74664085d84d9e750100ac50745733f6d8c0c66b8126d3aa96ff37"}
Nov 03 11:50:35.806702 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:35.806505    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="668ab7bfabcc6fb3ae8e48e8226ef1adb43820641df68271b6fdc3f5b9b4dbda" exitCode=0
Nov 03 11:50:35.807084 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:35.806586    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerDied","Data":"668ab7bfabcc6fb3ae8e48e8226ef1adb43820641df68271b6fdc3f5b9b4dbda"}
Nov 03 11:50:36.673245 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:36.673217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:36.673359 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:36.673300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:36.673359 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:36.673345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:36.673477 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:36.673378    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:36.673477 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:36.673433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:36.673581 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:36.673515    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:36.812998 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:36.812961    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"cad8d3cfd9cebb4a35facf22911fd88c1799a875817949b2597e8ae33eab72be"}
Nov 03 11:50:37.818355 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:37.818324    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerStarted","Data":"7eaccfa6769108696e8ed2098a6c8c0549e3c6fcb1aae41703498f23f500f2bc"}
Nov 03 11:50:38.672376 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.672338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:38.672376 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.672361    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:38.672607 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.672442    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:38.672607 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:38.672450    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:38.672607 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:38.672509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:38.672719 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:38.672607    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:38.823717 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.823684    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="7eaccfa6769108696e8ed2098a6c8c0549e3c6fcb1aae41703498f23f500f2bc" exitCode=0
Nov 03 11:50:38.824389 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.823804    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerDied","Data":"7eaccfa6769108696e8ed2098a6c8c0549e3c6fcb1aae41703498f23f500f2bc"}
Nov 03 11:50:38.828452 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.828422    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" event={"ID":"45488046-69c7-4931-b7b7-6efe7be609df","Type":"ContainerStarted","Data":"927a8e8d83b630f88976317df97895b5a473bed578ac84e2545c18a3637148dc"}
Nov 03 11:50:38.828815 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.828791    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:38.828933 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.828895    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:38.851290 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.851260    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:38.851479 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:38.851417    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:39.059924 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:39.058490    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" podStartSLOduration=6.742127177 podStartE2EDuration="22.058472738s" totalImagesPullingTime="15.316345561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:39.056963517 +0000 UTC m=+22.877169167" watchObservedRunningTime="2026-11-03 11:50:39.058472738 +0000 UTC m=+22.878678377"
Nov 03 11:50:39.833708 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:39.833670    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="6bd467f016ea1c12cf08994db3978b8899370b8cbafb0c03a404edde4693c846" exitCode=0
Nov 03 11:50:39.834131 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:39.833785    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerDied","Data":"6bd467f016ea1c12cf08994db3978b8899370b8cbafb0c03a404edde4693c846"}
Nov 03 11:50:39.834131 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:39.833844    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 11:50:40.672103 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:40.672073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:40.672281 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:40.672127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:40.672281 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:40.672138    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:40.672281 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:40.672225    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:40.672394 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:40.672287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:40.672394 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:40.672349    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:40.839399 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:40.839368    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" event={"ID":"dd63d182-450e-45aa-a997-4c98efe630dc","Type":"ContainerStarted","Data":"b0c3d04a781f68a0eeae941bf8d635b25e68769d6b306638f4bfdc09e29087a7"}
Nov 03 11:50:40.839782 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:40.839440    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 11:50:40.951614 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:40.951507    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rq7tf" podStartSLOduration=8.347431157 podStartE2EDuration="23.951493844s" totalImagesPullingTime="15.604062687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:40.951283831 +0000 UTC m=+24.771489468" watchObservedRunningTime="2026-11-03 11:50:40.951493844 +0000 UTC m=+24.771699482"
Nov 03 11:50:41.504842 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:41.504793    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lpbww"]
Nov 03 11:50:41.505042 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:41.504920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:41.505042 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:41.505014    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:41.508473 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:41.508145    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgfhl"]
Nov 03 11:50:41.508473 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:41.508268    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:41.508473 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:41.508366    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:41.508767 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:41.508734    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s79nv"]
Nov 03 11:50:41.508850 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:41.508838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:41.508973 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:41.508954    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:42.672834 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:42.672801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:42.672834 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:42.672833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:42.673483 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:42.672921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:42.673483 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:42.673039    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:43.319584 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:43.319549    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:43.319858 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:43.319837    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 11:50:43.331487 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:43.331447    2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" podUID="45488046-69c7-4931-b7b7-6efe7be609df" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 11:50:43.340784 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:43.340730    2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm" podUID="45488046-69c7-4931-b7b7-6efe7be609df" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 11:50:43.672969 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:43.672878    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:43.673348 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:43.672982    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:44.672744 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:44.672712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:44.672940 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:44.672846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s79nv" podUID="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce"
Nov 03 11:50:44.672940 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:44.672927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:44.673253 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:44.673021    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lpbww" podUID="606502b8-aa74-4661-94fe-43d62bfb8d40"
Nov 03 11:50:45.672374 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:45.672187    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:45.672559 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:45.672450    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgfhl" podUID="62d43ee3-52a9-4c5f-80f1-81b795f04ea6"
Nov 03 11:50:46.527474 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.527409    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeReady"
Nov 03 11:50:46.527886 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.527586    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 11:50:46.634486 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.634444    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-22bn2"]
Nov 03 11:50:46.639800 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.639768    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c9tgf"]
Nov 03 11:50:46.639957 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.639880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.643973 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.643948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.672331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.672296    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:46.672331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.672327    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:46.692559 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.692519    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nxjhp\""
Nov 03 11:50:46.692559 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.692518    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 11:50:46.692559 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.692520    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 11:50:46.692559 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.692560    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 11:50:46.693386 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.693366    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 11:50:46.693631 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.693607    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-22bn2"]
Nov 03 11:50:46.693730 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.693635    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 11:50:46.724710 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.724680    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:50:46.724710 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.724688    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:50:46.724958 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.724805    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzgk2\""
Nov 03 11:50:46.739517 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.739492    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:50:46.739635 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.739597    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:50:46.756474 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.756443    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c9tgf"]
Nov 03 11:50:46.765945 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.765914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372ffe3a-8e0e-4e2e-ab32-074098ec0133-tmp-dir\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.766093 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.765987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.766205 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.766182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2znb\" (UniqueName: \"kubernetes.io/projected/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-kube-api-access-l2znb\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.766311 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.766272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/372ffe3a-8e0e-4e2e-ab32-074098ec0133-config-volume\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.766355 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.766316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-cert\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.766395 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.766363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9tj\" (UniqueName: \"kubernetes.io/projected/372ffe3a-8e0e-4e2e-ab32-074098ec0133-kube-api-access-fg9tj\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.867283 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-cert\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.867283 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9tj\" (UniqueName: \"kubernetes.io/projected/372ffe3a-8e0e-4e2e-ab32-074098ec0133-kube-api-access-fg9tj\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.867515 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372ffe3a-8e0e-4e2e-ab32-074098ec0133-tmp-dir\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.867515 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.867617 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:46.867595    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:46.867617 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2znb\" (UniqueName: \"kubernetes.io/projected/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-kube-api-access-l2znb\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.867704 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867625    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372ffe3a-8e0e-4e2e-ab32-074098ec0133-tmp-dir\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.867704 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:46.867659    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:47.367644281 +0000 UTC m=+31.187849897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:46.867704 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.867694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/372ffe3a-8e0e-4e2e-ab32-074098ec0133-config-volume\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.868187 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.868168    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/372ffe3a-8e0e-4e2e-ab32-074098ec0133-config-volume\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.872160 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.872133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-cert\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.922228 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.922192    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9tj\" (UniqueName: \"kubernetes.io/projected/372ffe3a-8e0e-4e2e-ab32-074098ec0133-kube-api-access-fg9tj\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:46.922390 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.922316    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2znb\" (UniqueName: \"kubernetes.io/projected/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-kube-api-access-l2znb\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:46.954153 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:46.954102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:47.331190 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.330777    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-22bn2"]
Nov 03 11:50:47.334476 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:47.334449    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b8fe32_ab27_46d3_bc8d_e562b27a82cf.slice/crio-40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7 WatchSource:0}: Error finding container 40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7: Status 404 returned error can't find the container with id 40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7
Nov 03 11:50:47.372251 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.372220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:47.372404 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:47.372341    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:47.372404 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:47.372395    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:48.37238329 +0000 UTC m=+32.192588907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:47.672328 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.672232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:47.722092 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.722049    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:50:47.722092 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.722069    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:50:47.722092 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.722049    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:50:47.854542 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:47.854504    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-22bn2" event={"ID":"88b8fe32-ab27-46d3-bc8d-e562b27a82cf","Type":"ContainerStarted","Data":"40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7"}
Nov 03 11:50:48.379919 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.379628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:48.380109 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:48.379813    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:48.380109 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:48.380066    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.38004374 +0000 UTC m=+34.200249358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:48.499623 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.499585    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"]
Nov 03 11:50:48.509831 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.509801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:48.536140 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.536101    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4c659"]
Nov 03 11:50:48.554953 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.554925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.581559 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.581529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6l8\" (UniqueName: \"kubernetes.io/projected/870c96d8-257f-439b-a780-6a4f53964f50-kube-api-access-cf6l8\") pod \"migrator-66df5d496b-wspzm\" (UID: \"870c96d8-257f-439b-a780-6a4f53964f50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:48.582848 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.582815    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:48.583633 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.583611    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"]
Nov 03 11:50:48.586339 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.586321    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 11:50:48.586339 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.586335    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fw7f8\""
Nov 03 11:50:48.612878 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.612845    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4c659"]
Nov 03 11:50:48.660119 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.660007    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 11:50:48.660262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.660118    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 11:50:48.660262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.660136    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 11:50:48.660262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.660229    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7npm\""
Nov 03 11:50:48.660262 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.660229    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 11:50:48.664769 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.664724    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gs7fz-8qhqm"]
Nov 03 11:50:48.682331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78648619-04c1-4830-ba9a-9b9f8f568550-data-volume\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.682331 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.682860 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.682860 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78648619-04c1-4830-ba9a-9b9f8f568550-crio-socket\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.682860 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6l8\" (UniqueName: \"kubernetes.io/projected/870c96d8-257f-439b-a780-6a4f53964f50-kube-api-access-cf6l8\") pod \"migrator-66df5d496b-wspzm\" (UID: \"870c96d8-257f-439b-a780-6a4f53964f50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:48.682860 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.682860 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.682645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd2k\" (UniqueName: \"kubernetes.io/projected/78648619-04c1-4830-ba9a-9b9f8f568550-kube-api-access-wxd2k\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.722151 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.722121    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gs7fz-8qhqm"]
Nov 03 11:50:48.737291 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.737265    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6l8\" (UniqueName: \"kubernetes.io/projected/870c96d8-257f-439b-a780-6a4f53964f50-kube-api-access-cf6l8\") pod \"migrator-66df5d496b-wspzm\" (UID: \"870c96d8-257f-439b-a780-6a4f53964f50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:48.760185 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.760153    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 11:50:48.760320 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.760281    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pf74h\""
Nov 03 11:50:48.783339 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.783339 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.783535 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78648619-04c1-4830-ba9a-9b9f8f568550-crio-socket\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.783535 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.783535 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd2k\" (UniqueName: \"kubernetes.io/projected/78648619-04c1-4830-ba9a-9b9f8f568550-kube-api-access-wxd2k\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.783535 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:48.783479    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:48.783694 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.783694 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:48.783587    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:49.283570772 +0000 UTC m=+33.103776388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:48.783694 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783641    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78648619-04c1-4830-ba9a-9b9f8f568550-crio-socket\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.783694 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:48.783672    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 11:50:48.783694 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.783962 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.783962 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:48.783770    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:49.283732052 +0000 UTC m=+33.103937671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : configmap "kube-rbac-proxy" not found
Nov 03 11:50:48.783962 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.783841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78648619-04c1-4830-ba9a-9b9f8f568550-data-volume\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.784078 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.784063    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78648619-04c1-4830-ba9a-9b9f8f568550-data-volume\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.829351 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.829318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:48.829783 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.829742    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 11:50:48.838001 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.837972    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd2k\" (UniqueName: \"kubernetes.io/projected/78648619-04c1-4830-ba9a-9b9f8f568550-kube-api-access-wxd2k\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:48.885616 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.885588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.885724 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.885632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.885724 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.885686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.885902 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.885743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.886528 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.886498    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.886853 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.886825    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.888781 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.887819    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:48.941396 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:48.941333    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:49.006725 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.006689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:49.195619 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.195558    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"]
Nov 03 11:50:49.198278 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:49.198255    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870c96d8_257f_439b_a780_6a4f53964f50.slice/crio-a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d WatchSource:0}: Error finding container a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d: Status 404 returned error can't find the container with id a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d
Nov 03 11:50:49.289448 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.289402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:49.289625 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.289501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:49.289625 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:49.289562    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:49.289625 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:49.289588    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 11:50:49.289741 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:49.289629    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.289615159 +0000 UTC m=+34.109820793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:49.289741 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:49.289651    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:50.289635462 +0000 UTC m=+34.109841098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : configmap "kube-rbac-proxy" not found
Nov 03 11:50:49.397632 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.397597    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gs7fz-8qhqm"]
Nov 03 11:50:49.447633 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.447565    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4st2x"]
Nov 03 11:50:49.469833 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.469805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.504415 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.504387    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4st2x"]
Nov 03 11:50:49.520361 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.520333    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 11:50:49.520542 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.520414    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 11:50:49.520542 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.520478    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 11:50:49.520656 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.520542    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7tbzl\""
Nov 03 11:50:49.520656 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.520610    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 11:50:49.591834 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.591803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-key\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.591834 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.591834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgk6\" (UniqueName: \"kubernetes.io/projected/6a115576-c68b-4963-a7bc-5cc5232602e6-kube-api-access-jwgk6\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.592162 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.591885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-cabundle\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.692676 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.692640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-key\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.692676 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.692680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgk6\" (UniqueName: \"kubernetes.io/projected/6a115576-c68b-4963-a7bc-5cc5232602e6-kube-api-access-jwgk6\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.693223 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.692832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-cabundle\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.693567 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.693541    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-cabundle\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.695602 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.695582    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-key\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.750193 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.750168    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgk6\" (UniqueName: \"kubernetes.io/projected/6a115576-c68b-4963-a7bc-5cc5232602e6-kube-api-access-jwgk6\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.791020 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.790988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:49.861475 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.861448    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm" event={"ID":"de76cb62-4bfe-40a0-8b30-2de34cb21dc9","Type":"ContainerStarted","Data":"3e1672a39e89fee7b509946c07949a123194ef8ad1247d5e08f96676a746940d"}
Nov 03 11:50:49.862620 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:49.862597    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm" event={"ID":"870c96d8-257f-439b-a780-6a4f53964f50","Type":"ContainerStarted","Data":"a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d"}
Nov 03 11:50:50.163557 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.163488    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4st2x"]
Nov 03 11:50:50.166351 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:50.166326    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a115576_c68b_4963_a7bc_5cc5232602e6.slice/crio-9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f WatchSource:0}: Error finding container 9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f: Status 404 returned error can't find the container with id 9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f
Nov 03 11:50:50.297506 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.297470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:50.297688 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.297567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:50.297688 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.297668    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 11:50:50.297822 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.297692    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:50.297822 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.297743    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.297724505 +0000 UTC m=+36.117930124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : configmap "kube-rbac-proxy" not found
Nov 03 11:50:50.297921 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.297846    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.297837551 +0000 UTC m=+36.118043167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:50.398904 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.398636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:50.398904 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.398684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:50.398904 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.398787    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:50.398904 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.398835    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:50.398904 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.398862    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:54.398842401 +0000 UTC m=+38.219048022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:50.398904 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:50.398885    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:51:22.398869077 +0000 UTC m=+66.219074697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : secret "metrics-daemon-secret" not found
Nov 03 11:50:50.499826 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.499797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:50.500013 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.499847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:50.502526 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.502501    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:50.502792 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.502771    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:50.593282 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.593247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:50.744723 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.744693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:50.866189 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.866146    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x" event={"ID":"6a115576-c68b-4963-a7bc-5cc5232602e6","Type":"ContainerStarted","Data":"9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f"}
Nov 03 11:50:50.979414 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:50.979320    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lpbww"]
Nov 03 11:50:50.990986 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:50.990945    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606502b8_aa74_4661_94fe_43d62bfb8d40.slice/crio-e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019 WatchSource:0}: Error finding container e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019: Status 404 returned error can't find the container with id e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019
Nov 03 11:50:51.132608 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:51.132579    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgfhl"]
Nov 03 11:50:51.146826 ip-10-0-136-245 kubenswrapper[2582]: W1103 11:50:51.146792    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d43ee3_52a9_4c5f_80f1_81b795f04ea6.slice/crio-04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06 WatchSource:0}: Error finding container 04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06: Status 404 returned error can't find the container with id 04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06
Nov 03 11:50:51.870455 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:51.870373    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lpbww" event={"ID":"606502b8-aa74-4661-94fe-43d62bfb8d40","Type":"ContainerStarted","Data":"e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019"}
Nov 03 11:50:51.872674 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:51.872646    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm" event={"ID":"870c96d8-257f-439b-a780-6a4f53964f50","Type":"ContainerStarted","Data":"f1814016fa465183502a0e23698746d087da635e15ea5be469fd4bb98edd9e12"}
Nov 03 11:50:51.874498 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:51.874471    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgfhl" event={"ID":"62d43ee3-52a9-4c5f-80f1-81b795f04ea6","Type":"ContainerStarted","Data":"04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06"}
Nov 03 11:50:51.982177 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:51.982119    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm" podStartSLOduration=1.6410802580000001 podStartE2EDuration="3.982104675s" totalImagesPullingTime="2.341024417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:51.98182258 +0000 UTC m=+35.802028219" watchObservedRunningTime="2026-11-03 11:50:51.982104675 +0000 UTC m=+35.802310313"
Nov 03 11:50:52.314483 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:52.314447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:52.314656 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:52.314511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:52.314656 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:52.314614    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:52.314771 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:52.314688    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:56.3146703 +0000 UTC m=+40.134875921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:52.315085 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:52.315066    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:52.879971 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:52.879925    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-22bn2" event={"ID":"88b8fe32-ab27-46d3-bc8d-e562b27a82cf","Type":"ContainerStarted","Data":"8332a1b7f5c6a8781d894d8b9dc8027d1711e2f548ae30e233a90fd8acba9640"}
Nov 03 11:50:53.005886 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:53.005272    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-22bn2" podStartSLOduration=1.9862551179999999 podStartE2EDuration="7.005257946s" totalImagesPullingTime="5.019002828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:53.005016054 +0000 UTC m=+36.825221693" watchObservedRunningTime="2026-11-03 11:50:53.005257946 +0000 UTC m=+36.825463584"
Nov 03 11:50:54.431705 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:54.431607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:54.432081 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:54.431812    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:54.432081 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:54.431879    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:02.431865122 +0000 UTC m=+46.252070739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:54.887927 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:54.887882    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm" event={"ID":"de76cb62-4bfe-40a0-8b30-2de34cb21dc9","Type":"ContainerStarted","Data":"be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832"}
Nov 03 11:50:54.890742 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:54.890713    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x" event={"ID":"6a115576-c68b-4963-a7bc-5cc5232602e6","Type":"ContainerStarted","Data":"3479e02411d73f7f0b3a93187a8b238601f868d11a04fc9eb947c0e745ec5a40"}
Nov 03 11:50:54.999719 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:54.999656    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm" podStartSLOduration=2.582834929 podStartE2EDuration="6.999638313s" totalImagesPullingTime="4.416803384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:54.998872098 +0000 UTC m=+38.819077739" watchObservedRunningTime="2026-11-03 11:50:54.999638313 +0000 UTC m=+38.819843951"
Nov 03 11:50:55.107453 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:55.107390    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x" podStartSLOduration=2.098113768 podStartE2EDuration="6.107328136s" totalImagesPullingTime="4.009214368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:55.106651419 +0000 UTC m=+38.926857054" watchObservedRunningTime="2026-11-03 11:50:55.107328136 +0000 UTC m=+38.927533778"
Nov 03 11:50:56.352103 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:56.352056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:56.352541 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:56.352250    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:56.352541 ip-10-0-136-245 kubenswrapper[2582]: E1103 11:50:56.352323    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls podName:78648619-04c1-4830-ba9a-9b9f8f568550 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:04.352303727 +0000 UTC m=+48.172509347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c659" (UID: "78648619-04c1-4830-ba9a-9b9f8f568550") : secret "insights-runtime-extractor-tls" not found
Nov 03 11:50:57.900580 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:57.900535    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgfhl" event={"ID":"62d43ee3-52a9-4c5f-80f1-81b795f04ea6","Type":"ContainerStarted","Data":"20e87c4ec52b00201025da6886577dce9d18966148bae452ab4653076e768942"}
Nov 03 11:50:57.901011 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:57.900710    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:58.009739 ip-10-0-136-245 kubenswrapper[2582]: I1103 11:50:58.009691    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pgfhl" podStartSLOduration=34.701835709 podStartE2EDuration="41.009678287s" totalImagesPullingTime="6.307842578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:58.009462718 +0000 UTC m=+41.829668357" watchObservedRunningTime="2026-11-03 11:50:58.009678287 +0000 UTC m=+41.829883945"
Nov 03 11:50:58.361567 ip-10-0-136-245 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 11:50:58.368542 ip-10-0-136-245 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 11:50:58.368707 ip-10-0-136-245 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 11:50:58.368932 ip-10-0-136-245 systemd[1]: kubelet.service: Consumed 2.540s CPU time.
Nov 03 11:50:58.379504 ip-10-0-136-245 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 11:50:58.486078 ip-10-0-136-245 kubenswrapper[4546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:58.486078 ip-10-0-136-245 kubenswrapper[4546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 11:50:58.486078 ip-10-0-136-245 kubenswrapper[4546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:58.486078 ip-10-0-136-245 kubenswrapper[4546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:58.488181 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488096    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.488181 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488181    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.488181 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488185    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488188    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488191    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488194    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488197    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488199    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488202    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488204    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488207    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488209    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488212    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488215    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488217    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488220    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488222    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488225    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488228    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488231    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488233    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488236    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.488288 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488238    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488241    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488244    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488247    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488249    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488253    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488255    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488258    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488261    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488263    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488266    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488269    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488272    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488275    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488288    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488291    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488295    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488298    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488300    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488303    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.488789 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488306    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488308    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488311    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488314    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488316    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488319    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488321    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488324    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488326    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488329    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488331    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488334    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488336    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488339    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488342    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488345    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488347    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488351    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488354    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488356    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.489305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488359    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488361    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488365    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488369    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488373    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488375    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488378    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488381    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488384    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488387    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488389    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488394    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488398    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488401    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488405    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488408    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488411    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488414    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.489853 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488416    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488419    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488421    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488424    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488427    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488429    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488432    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488435    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488437    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488440    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488442    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488445    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488447    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488450    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488452    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488454    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488457    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488460    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488462    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.490310 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488465    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488467    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488470    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488473    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488475    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488478    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.488480    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489261    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489268    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489271    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489274    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489276    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489279    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489281    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489285    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489287    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489290    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489292    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489295    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489297    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.490838 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489300    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489302    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489305    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489308    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489311    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489314    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489316    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489319    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489322    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489324    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489327    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489329    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489332    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489334    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489337    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489339    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489347    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489350    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489353    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489356    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.491327 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489359    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489362    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489364    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489367    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489369    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489372    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489374    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489377    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489379    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489382    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489384    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489388    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489390    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489393    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489395    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489397    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489400    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489402    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489404    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.491835 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489407    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489409    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489412    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489414    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489416    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489419    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489421    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489424    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489426    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489429    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489432    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489435    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489438    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489442    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489445    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489448    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489450    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489453    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489455    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489457    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.492335 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489460    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489462    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489465    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489467    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489470    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489472    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489475    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489479    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489484    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489488    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489491    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489493    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489496    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489498    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489501    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489503    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489506    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489508    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489511    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.492856 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489513    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489516    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489518    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489521    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489524    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489528    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489532    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489534    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489537    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489539    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489542    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489544    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489547    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489549    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.489552    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490095    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490101    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490103    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490106    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.493334 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490109    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490112    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490114    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490118    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490121    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490124    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490127    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490132    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490134    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490137    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490139    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490142    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490145    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490147    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490150    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490152    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490155    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490162    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.493804 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490166    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490170    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490174    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490176    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490179    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490182    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490185    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490188    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490192    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490194    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490198    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490200    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490203    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490205    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490208    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490210    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490213    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490215    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490217    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490220    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.494261 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490222    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490226    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490229    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490232    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490234    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490237    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490239    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490242    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490244    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490248    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490250    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490253    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490256    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490259    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490261    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490264    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490266    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490269    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490271    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490274    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.494960 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490276    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490279    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490282    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490284    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490286    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490289    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490291    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490294    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490296    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490299    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490301    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490303    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490306    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490310    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490312    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490315    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490317    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490320    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490322    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490325    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.495494 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490328    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490331    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490333    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490336    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490338    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490347    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490349    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490352    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490354    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490357    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490359    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490362    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490364    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490367    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490369    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490371    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490374    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490377    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490380    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490382    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.496305 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490385    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490387    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490390    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.490392    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490498    4546 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490513    4546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490522    4546 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490526    4546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490530    4546 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490534    4546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490538    4546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490542    4546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490545    4546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490548    4546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490552    4546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490555    4546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490558    4546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490560    4546 flags.go:64] FLAG: --cgroup-root=""
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490571    4546 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490574    4546 flags.go:64] FLAG: --client-ca-file=""
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490577    4546 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490579    4546 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490586    4546 flags.go:64] FLAG: --cluster-domain=""
Nov 03 11:50:58.497077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490589    4546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490594    4546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490597    4546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490600    4546 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490604    4546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490607    4546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490610    4546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490614    4546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490617    4546 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490620    4546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490622    4546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490626    4546 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490629    4546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490633    4546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490638    4546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490640    4546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490643    4546 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490646    4546 flags.go:64] FLAG: --enable-server="true"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490649    4546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490656    4546 flags.go:64] FLAG: --event-burst="100"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490659    4546 flags.go:64] FLAG: --event-qps="50"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490662    4546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490665    4546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490668    4546 flags.go:64] FLAG: --eviction-hard=""
Nov 03 11:50:58.497632 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490672    4546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490675    4546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490678    4546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490681    4546 flags.go:64] FLAG: --eviction-soft=""
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490690    4546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490693    4546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490695    4546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490698    4546 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490701    4546 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490705    4546 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490708    4546 flags.go:64] FLAG: --feature-gates=""
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490712    4546 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490715    4546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490719    4546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490722    4546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490725    4546 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490727    4546 flags.go:64] FLAG: --help="false"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490730    4546 flags.go:64] FLAG: --hostname-override="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490733    4546 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490736    4546 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490739    4546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490742    4546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490761    4546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 11:50:58.498333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490764    4546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490767    4546 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490770    4546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490773    4546 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490775    4546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490779    4546 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490782    4546 flags.go:64] FLAG: --kube-reserved=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490785    4546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490787    4546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490790    4546 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490793    4546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490796    4546 flags.go:64] FLAG: --lock-file=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490799    4546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490801    4546 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490810    4546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490816    4546 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490819    4546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490822    4546 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490825    4546 flags.go:64] FLAG: --logging-format="text"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490828    4546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490832    4546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490835    4546 flags.go:64] FLAG: --manifest-url=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490837    4546 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490842    4546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490845    4546 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 11:50:58.498946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490849    4546 flags.go:64] FLAG: --max-pods="110"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490852    4546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490855    4546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490858    4546 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490861    4546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490864    4546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490867    4546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490870    4546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490877    4546 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490880    4546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490883    4546 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490886    4546 flags.go:64] FLAG: --pod-cidr=""
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490889    4546 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490892    4546 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490895    4546 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490898    4546 flags.go:64] FLAG: --port="10250"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490900    4546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490903    4546 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01f9330680dc2b7c4"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490906    4546 flags.go:64] FLAG: --qos-reserved=""
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490909    4546 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490912    4546 flags.go:64] FLAG: --register-node="true"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490915    4546 flags.go:64] FLAG: --register-with-taints=""
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490924    4546 flags.go:64] FLAG: --registry-burst="10"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490927    4546 flags.go:64] FLAG: --registry-qps="5"
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490930    4546 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490938    4546 flags.go:64] FLAG: --reserved-memory=""
Nov 03 11:50:58.499536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490942    4546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490945    4546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490948    4546 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490951    4546 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490954    4546 flags.go:64] FLAG: --runonce="false"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490957    4546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490960    4546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490963    4546 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490966    4546 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490968    4546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490971    4546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490974    4546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490978    4546 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490981    4546 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490984    4546 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490987    4546 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490990    4546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490993    4546 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490996    4546 flags.go:64] FLAG: --system-cgroups=""
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.490999    4546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491004    4546 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491007    4546 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491010    4546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491021    4546 flags.go:64] FLAG: --tls-min-version=""
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491024    4546 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 11:50:58.500344 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491027    4546 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491030    4546 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491033    4546 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491036    4546 flags.go:64] FLAG: --v="2"
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491045    4546 flags.go:64] FLAG: --version="false"
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491050    4546 flags.go:64] FLAG: --vmodule=""
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491056    4546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491060    4546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491237    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491240    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491243    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491245    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491248    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491251    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491253    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491255    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491258    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491260    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491263    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491268    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491271    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.500985 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491273    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491276    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491278    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491281    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491284    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491286    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491289    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491291    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491294    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491296    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491299    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491301    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491304    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491306    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491309    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491312    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491320    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491324    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491327    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.501572 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491329    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491332    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491335    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491337    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491341    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491345    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491348    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491350    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491353    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491355    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491358    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491362    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491366    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491368    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491371    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491374    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491376    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491379    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491381    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.502119 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491383    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491386    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491388    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491391    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491393    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491396    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491398    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491401    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491403    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491406    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491408    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491417    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491420    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491422    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491425    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491427    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491430    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491433    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491435    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491438    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.502830 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491440    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491443    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491445    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491448    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491450    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491453    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491455    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491458    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491460    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491463    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491465    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491467    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491470    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491472    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491475    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491477    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491480    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491482    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491485    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491487    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.503400 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491489    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491492    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491494    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491498    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491506    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491509    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491511    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491514    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491516    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491519    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491521    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491524    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491526    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491529    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491531    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.491557    4546 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:58.504268 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491603    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491607    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491611    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491613    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491616    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491619    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491622    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491625    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491628    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491632    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491635    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491638    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491641    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491643    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491646    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491650    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491654    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491657    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491660    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.504966 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491664    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491666    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491675    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491678    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491681    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491683    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491686    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491688    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491691    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491693    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491696    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491700    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491702    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491705    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491708    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491710    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491712    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491715    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491717    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491719    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.505510 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491722    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491724    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491727    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491729    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491732    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491734    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491737    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491740    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491743    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491758    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491761    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491763    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491768    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491771    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491773    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491781    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491784    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491786    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491789    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491791    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.506132 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491794    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491796    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491799    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491801    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491804    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491807    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491809    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491812    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491814    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491817    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491819    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491822    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491824    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491827    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491829    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491832    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491834    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491837    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491840    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491844    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.506659 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491846    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491849    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491851    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491854    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491857    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491860    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491862    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491865    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491873    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491876    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491879    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491881    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491884    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491886    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491889    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491891    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491894    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491896    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491899    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.507170 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491901    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491903    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491906    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491908    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491911    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491913    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491916    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.491918    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.497184    4546 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.497205    4546 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.497230    4546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.497236    4546 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497300    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497306    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497310    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497313    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497316    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497319    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497322    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.507667 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497324    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497327    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497329    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497332    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497334    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497337    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497339    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497342    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497344    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497346    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497349    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497351    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497354    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497357    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497359    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497362    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497364    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497367    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497369    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497372    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.508223 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497375    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497377    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497380    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497382    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497385    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497387    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497390    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497393    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497395    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497398    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497400    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497403    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497406    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497408    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497410    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497413    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497415    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497417    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497420    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497432    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.508731 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497434    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497437    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497439    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497441    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497444    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497446    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497449    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497451    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497454    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497456    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497459    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497462    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497466    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497469    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497471    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497474    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497476    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497479    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497481    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497484    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.509346 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497487    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497489    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497492    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497494    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497497    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497499    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497502    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497504    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497506    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497509    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497512    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497514    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497523    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497525    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497528    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497531    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497533    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497536    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497538    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.509844 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497541    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497543    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497547    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497551    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497554    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497556    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497560    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497562    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497565    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497567    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497570    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497572    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497575    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497577    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497580    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497583    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497585    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497588    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497590    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.510296 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497593    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.497622    4546 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497771    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497776    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497780    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497783    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497786    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497794    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497797    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497800    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497802    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497805    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497807    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497810    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497815    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497818    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.510773 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497820    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497823    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497825    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497828    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497830    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497833    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497836    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497838    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497841    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497843    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497846    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497849    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497851    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497854    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497856    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497859    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497862    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497864    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497867    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.511176 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497869    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497872    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497874    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497877    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497879    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497887    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497890    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497893    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497895    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497898    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497901    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497906    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497910    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497912    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497915    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497918    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497920    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497923    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497925    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497927    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.511636 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497930    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497933    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497935    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497938    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497941    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497943    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497946    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497948    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497951    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497953    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497956    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497959    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497961    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497964    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497966    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497969    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497971    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497973    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497981    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.512188 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497984    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497986    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497989    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497991    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497995    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.497999    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498001    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498004    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498006    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498010    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498014    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498017    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498019    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498021    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498024    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498026    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498029    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498031    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498034    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.512669 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498036    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498039    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498041    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498044    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498046    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498049    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498051    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498054    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498056    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498059    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498061    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498063    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498066    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498073    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498076    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.498101    4546 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:58.513149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498133    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498137    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498143    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498146    4546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498149    4546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498151    4546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498154    4546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498156    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498159    4546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498161    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498163    4546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498166    4546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498168    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498171    4546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498174    4546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498176    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498179    4546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498181    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:58.513550 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498185    4546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498189    4546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498192    4546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498194    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498197    4546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498199    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498202    4546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498204    4546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498207    4546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498209    4546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498212    4546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498214    4546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498222    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498225    4546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498227    4546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498230    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498235    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498238    4546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498240    4546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:58.514032 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498243    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498245    4546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498248    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498250    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498253    4546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498255    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498258    4546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498260    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498263    4546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498266    4546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498268    4546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498271    4546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498273    4546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498276    4546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498278    4546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498281    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498283    4546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498286    4546 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498288    4546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498291    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:58.514487 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498293    4546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498296    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498298    4546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498301    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498303    4546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498306    4546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498317    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498319    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498322    4546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498326    4546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498329    4546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498331    4546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498334    4546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498337    4546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498339    4546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498342    4546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498345    4546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498347    4546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498350    4546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:58.514986 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498353    4546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498355    4546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498358    4546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498360    4546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498363    4546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498365    4546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498368    4546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498371    4546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498373    4546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498376    4546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498380    4546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498384    4546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498387    4546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498390    4546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498393    4546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498395    4546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498398    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498401    4546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498403    4546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498406    4546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:58.515455 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498415    4546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498417    4546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498422    4546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498424    4546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498427    4546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498429    4546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498431    4546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498434    4546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498436    4546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:50:58.498439    4546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.498496    4546 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.499431    4546 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.499494    4546 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.499683    4546 server.go:1014] "Starting client certificate rotation"
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.499857    4546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.499913    4546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 11:45:17 +0000 UTC" deadline="2028-04-08 14:54:34.736528609 +0000 UTC"
Nov 03 11:50:58.516005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.499954    4546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12531h3m36.236578479s"
Nov 03 11:50:58.516443 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.500196    4546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:58.516443 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.501290    4546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:58.516443 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.502977    4546 log.go:25] "Validated CRI v1 runtime API"
Nov 03 11:50:58.516443 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.505055    4546 log.go:25] "Validated CRI v1 image API"
Nov 03 11:50:58.516443 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.505794    4546 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 11:50:58.516443 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.507397    4546 fs.go:135] Filesystem UUIDs: map[4a323807-9fcb-4fcd-878d-a15fad11ee78:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c6ff839c-40e2-4202-9467-71074f89b07b:/dev/nvme0n1p3]
Nov 03 11:50:58.516610 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.507412    4546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e1672a39e89fee7b509946c07949a123194ef8ad1247d5e08f96676a746940d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e1672a39e89fee7b509946c07949a123194ef8ad1247d5e08f96676a746940d/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ead3a7f-804d-48a7-9dae-ec5df206cbbe/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2ead3a7f-804d-48a7-9dae-ec5df206cbbe/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/372ffe3a-8e0e-4e2e-ab32-074098ec0133/volumes/kubernetes.io~projected/kube-api-access-fg9tj:{mountpoint:/var/lib/kubelet/pods/372ffe3a-8e0e-4e2e-ab32-074098ec0133/volumes/kubernetes.io~projected/kube-api-access-fg9tj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volumes/kubernetes.io~projected/kube-api-access-56tkk:{mountpoint:/var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volumes/kubernetes.io~projected/kube-api-access-56tkk major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/606502b8-aa74-4661-94fe-43d62bfb8d40/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/606502b8-aa74-4661-94fe-43d62bfb8d40/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62d43ee3-52a9-4c5f-80f1-81b795f04ea6/volumes/kubernetes.io~projected/kube-api-access-tknkk:{mountpoint:/var/lib/kubelet/pods/62d43ee3-52a9-4c5f-80f1-81b795f04ea6/volumes/kubernetes.io~projected/kube-api-access-tknkk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a115576-c68b-4963-a7bc-5cc5232602e6/volumes/kubernetes.io~projected/kube-api-access-jwgk6:{mountpoint:/var/lib/kubelet/pods/6a115576-c68b-4963-a7bc-5cc5232602e6/volumes/kubernetes.io~projected/kube-api-access-jwgk6 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a115576-c68b-4963-a7bc-5cc5232602e6/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6a115576-c68b-4963-a7bc-5cc5232602e6/volumes/kubernetes.io~secret/signing-key major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78648619-04c1-4830-ba9a-9b9f8f568550/volumes/kubernetes.io~projected/kube-api-access-wxd2k:{mountpoint:/var/lib/kubelet/pods/78648619-04c1-4830-ba9a-9b9f8f568550/volumes/kubernetes.io~projected/kube-api-access-wxd2k major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/870c96d8-257f-439b-a780-6a4f53964f50/volumes/kubernetes.io~projected/kube-api-access-cf6l8:{mountpoint:/var/lib/kubelet/pods/870c96d8-257f-439b-a780-6a4f53964f50/volumes/kubernetes.io~projected/kube-api-access-cf6l8 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc/volumes/kubernetes.io~projected/kube-api-access-mv992:{mountpoint:/var/lib/kubelet/pods/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc/volumes/kubernetes.io~projected/kube-api-access-mv992 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88b8fe32-ab27-46d3-bc8d-e562b27a82cf/volumes/kubernetes.io~projected/kube-api-access-l2znb:{mountpoint:/var/lib/kubelet/pods/88b8fe32-ab27-46d3-bc8d-e562b27a82cf/volumes/kubernetes.io~projected/kube-api-access-l2znb major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88b8fe32-ab27-46d3-bc8d-e562b27a82cf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/88b8fe32-ab27-46d3-bc8d-e562b27a82cf/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9bee7ca-d883-4a3b-94c7-9ad031cad22f/volumes/kubernetes.io~projected/kube-api-access-7rcjz:{mountpoint:/var/lib/kubelet/pods/a9bee7ca-d883-4a3b-94c7-9ad031cad22f/volumes/kubernetes.io~projected/kube-api-access-7rcjz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b075924a-500c-4f50-ab6c-dbb51839181c/volumes/kubernetes.io~projected/kube-api-access-gpccs:{mountpoint:/var/lib/kubelet/pods/b075924a-500c-4f50-ab6c-dbb51839181c/volumes/kubernetes.io~projected/kube-api-access-gpccs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce/volumes/kubernetes.io~projected/kube-api-access-57x2n:{mountpoint:/var/lib/kubelet/pods/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce/volumes/kubernetes.io~projected/kube-api-access-57x2n major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c67983f1-d777-4fde-a01b-24d8a2a4f3c5/volumes/kubernetes.io~projected/kube-api-access-t5k9m:{mountpoint:/var/lib/kubelet/pods/c67983f1-d777-4fde-a01b-24d8a2a4f3c5/volumes/kubernetes.io~projected/kube-api-access-t5k9m major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd63d182-450e-45aa-a997-4c98efe630dc/volumes/kubernetes.io~projected/kube-api-access-4z49h:{mountpoint:/var/lib/kubelet/pods/dd63d182-450e-45aa-a997-4c98efe630dc/volumes/kubernetes.io~projected/kube-api-access-4z49h major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de76cb62-4bfe-40a0-8b30-2de34cb21dc9/volumes/kubernetes.io~projected/kube-api-access-q268p:{mountpoint:/var/lib/kubelet/pods/de76cb62-4bfe-40a0-8b30-2de34cb21dc9/volumes/kubernetes.io~projected/kube-api-access-q268p major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5e9caab-b867-45ec-b81c-b74d1821302d/volumes/kubernetes.io~projected/kube-api-access-ntspk:{mountpoint:/var/lib/kubelet/pods/e5e9caab-b867-45ec-b81c-b74d1821302d/volumes/kubernetes.io~projected/kube-api-access-ntspk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~projected/kube-api-access-hf7lm:{mountpoint:/var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~projected/kube-api-access-hf7lm major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/46aa206562f895fe5f42e7a911bfd953f2b8697fa02f703e474e9c5b3da938b1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/22b603bff9c7f65ac5199625fa42c0fdecd194b95135577753a5c6082c69ab8c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/4ee6863acc77c4cd50120e3b0d2061146b0db1c99b8ed05c0559b070c907c0f9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/d766cb18d93c8c65adef53fde60e154a240f946746ca9aba7e1feee3b34ce5d0/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/f3c642ba05039885e0d8c468b028bfe86a67086e1be4d67f1bdba29e76e6e3f8/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8f2a305de65ce7df8f29bb8c0df94cb31c8b11a155df9b52c21276a67f8381f9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/605ec61bb3bbb95f1097c37135ee2a6b550c645a0d6c12e49a95e2ac848ca27a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/bbf225f6946ec8b81c425d589c10c4e71e840ec55c4104f5d2f3dd634cf05152/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/98d439265082777d48ccd92095b9bd22c934e879516c5954fa8997621d70e35c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/923c9d9b21a01df345ce98f6730490794cac7061fc8db333f8f921957b867e32/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c1ebe03ddb3710f79c29fb9141d29630cc050bcdb0f8941cdecf58628fa4abd3/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/423cbf62abc52da2be1649bedcd9ad1536d141e0878c4fe3bca2196c6ae36e4b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/fc0d2bdf7542a0639c320dce21a9320ff43c38f82481c149d3676dccbf6ce033/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b1979dd48a558d4064380680f766ac584df383ec80f738a46a7079d4a925bce1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9d04b61b86e9bb376772ea9f05b384f4fd5124a7409fa7e0cd3a03fd9c240c6c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2e5be94b962347127137ada68f411d41e2d3bec096dc2b06243f926498744339/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e031dce45688451955e248d5788e943402b9649d8508394aa62a1d523aee8896/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bde351d1bd90cb4278cd4657e25861273759b1810c97d2846d7b4ec99922301d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3bf6266763b2b619527f4e925e9ffe9cc575f1e5617c56cf218cdf7103cb53e2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4e0402d6bb7ece94329dda06abf17e46a4213d2c8183bd05cdd265cd6b436375/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f1b95cc328221c6cf197bb3cefaf9395e284b0dc85529163bcbf57e46094bd2c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/9c6592fd9eb250ce870762f22306acdcb9e4050f9d6521f7bc8a91b4f541e57d/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ccd07c912053cc27fdc6bd9bb90b932f372596a00886fbae8ef6da7327dde1a5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/3d4062a7b04135d33ea2c3ad6519e9c3def6fcfafcf0ff4fc1cd410ee1acbe65/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/463eda1ab7f60daeb54730a533fe8bdd98a8e3ee757a56f64ada8493e0a34167/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/34ee3c3a6c5d3c6983cea37ce5ae4d6e56d4e4f9390e97796729fc30f3f84f7b/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/0a07e0bf51769ea38bf03abeafb294fcf7888d5d67060e00d0eaaf50010be024/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/a060b61cb8662d4ac9ae43de267dd86918040be778befb13850ac1af831ef4c7/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/28f97608ea310c1c60ada1044b3f336d0d459ff41709ce5263817fdce447511e/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e75dff154a99dfebe144b15f8436de9a98c03e3a72a824a6efa092be559518ae/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d4fadbcbedbc65329769c68c4644d4bd83dc8db8fc060a8eb41d926ad8954601/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/451b0be0f3b52d132d8ed88630f63368dd5058d6bc502ca506e59f409f73d8e6/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/813d7c2411f6caf34c9e65e34dd34ecb5d16d9e445105b99c9823722aff7e9d8/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/00a5bc532ff0a741ef107a6bf15202d768f0b75887e0d3b2a82f3ba9e41300c7/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/101f7a30b8019b4443ef292de10d3145eca3b3a75e381659e1174f854792fd85/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d06ca64a3181673361618d4d6e7336294915d9df9bacc8d4edd521372b89cf08/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a7e794ecee2dd878a9f4ef66569fe91430655ddab392bec5db0a2a61b4edc27a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d52ed09ffe715f6cd615392e89dbb9e90854ad4e9865d9d54f51bc2a827339d9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/04208171a5ec4ec6a5230e31b3d181b3da28b43a9a071a82b49d6945784b84dc/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7d660fc8eb7c681b445f443971bbf8edced6111f870fdbe32acd171e94d4360f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/167783db1f0d389aa60eb9886c18e17451ee8dcaafe2333dfc41223eac45e5f7/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/35c91035897dce3ec054050442adb4844cb5e2e11d074ee734f7071c16ca3372/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b7bebdeb32c5b18137f2ac8ef4ace5000982944eb67cc165442003f66a51da62/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.514533    4546 manager.go:217] Machine: {Timestamp:2026-11-03 11:50:58.51407441 +0000 UTC m=+0.085658313 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101712 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c5987f70ed9866df0a7d9936b42c SystemUUID:ec28c598-7f70-ed98-66df-0a7d9936b42c BootID:9479e698-c905-4b6e-bccf-9516e5c48247 Filesystems:[{Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a86ed6b1e8228d1710ffbab70b833d3668f2e64a8446d2adb6ccaa45b8dc09c4/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a115576-c68b-4963-a7bc-5cc5232602e6/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5e9caab-b867-45ec-b81c-b74d1821302d/volumes/kubernetes.io~projected/kube-api-access-ntspk DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc/volumes/kubernetes.io~projected/kube-api-access-mv992 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/606502b8-aa74-4661-94fe-43d62bfb8d40/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e075e97d9c30596f1f836728c9639ee5d40f8f71285dac13e26f2b3c92ae62f/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/372ffe3a-8e0e-4e2e-ab32-074098ec0133/volumes/kubernetes.io~projected/kube-api-access-fg9tj DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78648619-04c1-4830-ba9a-9b9f8f568550/volumes/kubernetes.io~projected/kube-api-access-wxd2k DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9bee7ca-d883-4a3b-94c7-9ad031cad22f/volumes/kubernetes.io~projected/kube-api-access-7rcjz DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2f2b6e076b4e8e0f42a77f81be46f5182df290fdbedf867dfec6af1b9a656db/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a4fbb0ac95bedb27917739fd04c98fca16c740b8de7b2b5c5a42d7a544d80b4/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53f5a93029e1707b855881e2eca5ff7b1ba973e1fcfcb48c406c63a8b511c9aa/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0aed057d6fbe034f4aae0a32dc9f401711e1d788ccf51531819b584ee6117b2c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62d43ee3-52a9-4c5f-80f1-81b795f04ea6/volumes/kubernetes.io~projected/kube-api-access-tknkk DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd63d182-450e-45aa-a997-4c98efe630dc/volumes/kubernetes.io~projected/kube-api-access-4z49h DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c67983f1-d777-4fde-a01b-24d8a2a4f3c5/volumes/kubernetes.io~projected/kube-api-access-t5k9m DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce/volumes/kubernetes.io~projected/kube-api-access-57x2n DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50880f9ec6d12f88951abf00e301ce6d6f991b69bce96cf2a8c695f8590a8fe1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de76cb62-4bfe-40a0-8b30-2de34cb21dc9/volumes/kubernetes.io~projected/kube-api-access-q268p DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e8f7ad723354fed308bdac842603632858e75ba674a367109a99a04be0bd60f/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volumes/kubernetes.io~projected/kube-api-access-56tkk DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b075924a-500c-4f50-ab6c-dbb51839181c/volumes/kubernetes.io~projected/kube-api-access-gpccs DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88b8fe32-ab27-46d3-bc8d-e562b27a82cf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e97af85633c9556ec1188af6247a2e7682fa8b90979517a1c0004d1b71a44019/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/870c96d8-257f-439b-a780-6a4f53964f50/volumes/kubernetes.io~projected/kube-api-access-cf6l8 DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d250c89b83a7e41f9110191aa965a64766945794a4b19c23605c165a6cd9bfca/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6a115576-c68b-4963-a7bc-5cc5232602e6/volumes/kubernetes.io~projected/kube-api-access-jwgk6 DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45488046-69c7-4931-b7b7-6efe7be609df/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/88b8fe32-ab27-46d3-bc8d-e562b27a82cf/volumes/kubernetes.io~projected/kube-api-access-l2znb DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40021fd392ccff2ec88959d6671c72f44a87186fc4e2be6dc141d815e08b29f7/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e1672a39e89fee7b509946c07949a123194ef8ad1247d5e08f96676a746940d/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04388bec8696b3ec6559f4cce103d8bda16d57ecfac4c5c5e027622360323b06/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ead3a7f-804d-48a7-9dae-ec5df206cbbe/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fdbf6c89-33f3-48a2-aa96-16a26e336e34/volumes/kubernetes.io~projected/kube-api-access-hf7lm DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a50ecd7a7ab7df197492e556ad1b86fc1329793d43c31366af0cba3cf345d61d/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04388bec8696b3e MacAddress:1e:55:d1:1b:78:5a Speed:10000 Mtu:8901} {Name:3e1672a39e89fee MacAddress:a6:84:19:67:01:a8 Speed:10000 Mtu:8901} {Name:40021fd392ccff2 MacAddress:92:b1:ee:f5:3c:76 Speed:10000 Mtu:8901} {Name:9e8f7ad723354fe MacAddress:d6:9b:18:d4:b5:8b Speed:10000 Mtu:8901} {Name:a50ecd7a7ab7df1 MacAddress:f6:2e:f5:d2:6b:63 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d7:40:3d:af Speed:0 Mtu:9001} {Name:br-int MacAddress:de:f5:ef:71:d5:1c Speed:0 Mtu:8901} {Name:e97af85633c9556 MacAddress:4a:41:fd:25:25:e8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d7:40:3d:af Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:fe:69:15:91:83 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:07:ec:e8:21:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.514828    4546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.514872    4546 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515131    4546 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515155    4546 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-245.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515295    4546 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515303    4546 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515318    4546 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515332    4546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515419    4546 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515581    4546 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515699    4546 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515738    4546 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515770    4546 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515789    4546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515802    4546 kubelet.go:405] "Adding apiserver pod source"
Nov 03 11:50:58.516889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.515811    4546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 11:50:58.518427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.517859    4546 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 11:50:58.518427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.517880    4546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:58.518965 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.518945    4546 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 11:50:58.521071 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521052    4546 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 11:50:58.521071 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521068    4546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:58.521221 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521180    4546 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 11:50:58.521416 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521401    4546 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 11:50:58.521475 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521441    4546 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 11:50:58.521566 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521555    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 11:50:58.521619 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521575    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 11:50:58.521619 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521586    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 11:50:58.521619 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521595    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 11:50:58.521619 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521605    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 11:50:58.521619 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521615    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521625    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521633    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521646    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521655    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521676    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521692    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521731    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 11:50:58.521927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.521741    4546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 11:50:58.525320 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.525303    4546 server.go:1263] "Started kubelet"
Nov 03 11:50:58.525414 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.525361    4546 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 11:50:58.525642 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.525539    4546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 11:50:58.525742 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.525704    4546 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 11:50:58.525867 ip-10-0-136-245 systemd[1]: Started Kubernetes Kubelet.
Nov 03 11:50:58.526196 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.526060    4546 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.526544    4546 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528319    4546 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528339    4546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528430    4546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:19 +0000 UTC" deadline="2028-05-11 09:21:22.784235745 +0000 UTC"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528445    4546 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528449    4546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13317h30m24.255789446s"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528453    4546 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.528463    4546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530076    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88b8fe32-ab27-46d3-bc8d-e562b27a82cf" volumeName="kubernetes.io/secret/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-cert" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530102    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b075924a-500c-4f50-ab6c-dbb51839181c" volumeName="kubernetes.io/empty-dir/b075924a-500c-4f50-ab6c-dbb51839181c-tmp-dir" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530113    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" volumeName="kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530126    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdbf6c89-33f3-48a2-aa96-16a26e336e34" volumeName="kubernetes.io/projected/fdbf6c89-33f3-48a2-aa96-16a26e336e34-kube-api-access-hf7lm" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530137    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ead3a7f-804d-48a7-9dae-ec5df206cbbe" volumeName="kubernetes.io/secret/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-agent-certs" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530148    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce" volumeName="kubernetes.io/projected/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-kube-api-access-57x2n" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530159    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdbf6c89-33f3-48a2-aa96-16a26e336e34" volumeName="kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-tmp" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530171    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="372ffe3a-8e0e-4e2e-ab32-074098ec0133" volumeName="kubernetes.io/empty-dir/372ffe3a-8e0e-4e2e-ab32-074098ec0133-tmp-dir" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530184    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" volumeName="kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530196    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a115576-c68b-4963-a7bc-5cc5232602e6" volumeName="kubernetes.io/projected/6a115576-c68b-4963-a7bc-5cc5232602e6-kube-api-access-jwgk6" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530207    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ead3a7f-804d-48a7-9dae-ec5df206cbbe" volumeName="kubernetes.io/configmap/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-konnectivity-ca" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530218    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78648619-04c1-4830-ba9a-9b9f8f568550" volumeName="kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530229    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="870c96d8-257f-439b-a780-6a4f53964f50" volumeName="kubernetes.io/projected/870c96d8-257f-439b-a780-6a4f53964f50-kube-api-access-cf6l8" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530241    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9bee7ca-d883-4a3b-94c7-9ad031cad22f" volumeName="kubernetes.io/projected/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kube-api-access-7rcjz" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530254    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd63d182-450e-45aa-a997-4c98efe630dc" volumeName="kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530266    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" volumeName="kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530277    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e9caab-b867-45ec-b81c-b74d1821302d" volumeName="kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-daemon-config" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530288    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a115576-c68b-4963-a7bc-5cc5232602e6" volumeName="kubernetes.io/secret/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-key" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530299    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd63d182-450e-45aa-a997-4c98efe630dc" volumeName="kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530310    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e9caab-b867-45ec-b81c-b74d1821302d" volumeName="kubernetes.io/projected/e5e9caab-b867-45ec-b81c-b74d1821302d-kube-api-access-ntspk" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530321    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdbf6c89-33f3-48a2-aa96-16a26e336e34" volumeName="kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-tuned" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530333    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a115576-c68b-4963-a7bc-5cc5232602e6" volumeName="kubernetes.io/configmap/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-cabundle" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530344    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="372ffe3a-8e0e-4e2e-ab32-074098ec0133" volumeName="kubernetes.io/configmap/372ffe3a-8e0e-4e2e-ab32-074098ec0133-config-volume" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530355    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45488046-69c7-4931-b7b7-6efe7be609df" volumeName="kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530365    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45488046-69c7-4931-b7b7-6efe7be609df" volumeName="kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-config" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530375    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45488046-69c7-4931-b7b7-6efe7be609df" volumeName="kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-env-overrides" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530386    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62d43ee3-52a9-4c5f-80f1-81b795f04ea6" volumeName="kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530397    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87b8775a-e0e0-4cf6-9be5-9df4891d6ccc" volumeName="kubernetes.io/projected/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-kube-api-access-mv992" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530425    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd63d182-450e-45aa-a997-4c98efe630dc" volumeName="kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-binary-copy" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530437    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="372ffe3a-8e0e-4e2e-ab32-074098ec0133" volumeName="kubernetes.io/projected/372ffe3a-8e0e-4e2e-ab32-074098ec0133-kube-api-access-fg9tj" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530447    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88b8fe32-ab27-46d3-bc8d-e562b27a82cf" volumeName="kubernetes.io/projected/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-kube-api-access-l2znb" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530458    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b075924a-500c-4f50-ab6c-dbb51839181c" volumeName="kubernetes.io/projected/b075924a-500c-4f50-ab6c-dbb51839181c-kube-api-access-gpccs" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530471    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c67983f1-d777-4fde-a01b-24d8a2a4f3c5" volumeName="kubernetes.io/configmap/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-serviceca" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530482    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd63d182-450e-45aa-a997-4c98efe630dc" volumeName="kubernetes.io/projected/dd63d182-450e-45aa-a997-4c98efe630dc-kube-api-access-4z49h" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530493    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" volumeName="kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530503    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e9caab-b867-45ec-b81c-b74d1821302d" volumeName="kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-cni-binary-copy" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530515    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="606502b8-aa74-4661-94fe-43d62bfb8d40" volumeName="kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530526    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78648619-04c1-4830-ba9a-9b9f8f568550" volumeName="kubernetes.io/projected/78648619-04c1-4830-ba9a-9b9f8f568550-kube-api-access-wxd2k" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530536    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87b8775a-e0e0-4cf6-9be5-9df4891d6ccc" volumeName="kubernetes.io/configmap/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-iptables-alerter-script" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530547    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c67983f1-d777-4fde-a01b-24d8a2a4f3c5" volumeName="kubernetes.io/projected/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-kube-api-access-t5k9m" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530558    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45488046-69c7-4931-b7b7-6efe7be609df" volumeName="kubernetes.io/projected/45488046-69c7-4931-b7b7-6efe7be609df-kube-api-access-56tkk" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530568    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45488046-69c7-4931-b7b7-6efe7be609df" volumeName="kubernetes.io/secret/45488046-69c7-4931-b7b7-6efe7be609df-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530579    4546 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78648619-04c1-4830-ba9a-9b9f8f568550" volumeName="kubernetes.io/empty-dir/78648619-04c1-4830-ba9a-9b9f8f568550-data-volume" seLinuxMountContext=""
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530591    4546 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.530598    4546 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.530122    4546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.531332    4546 factory.go:55] Registering systemd factory
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.531349    4546 factory.go:223] Registration of the systemd container factory successfully
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.532600    4546 factory.go:153] Registering CRI-O factory
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.532611    4546 factory.go:223] Registration of the crio container factory successfully
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.532649    4546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.532663    4546 factory.go:103] Registering Raw factory
Nov 03 11:50:58.533433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.532673    4546 manager.go:1199] Started watching for new ooms in manager
Nov 03 11:50:58.539792 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.538199    4546 manager.go:319] Starting recovery of all containers
Nov 03 11:50:58.563045 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.562997    4546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 11:50:58.566432 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.565019    4546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 11:50:58.566432 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.565048    4546 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 11:50:58.566432 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.565071    4546 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 11:50:58.566432 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.565110    4546 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 11:50:58.582186 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.582158    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="6bd467f016ea1c12cf08994db3978b8899370b8cbafb0c03a404edde4693c846" exitCode=0
Nov 03 11:50:58.582186 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.582181    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="7eaccfa6769108696e8ed2098a6c8c0549e3c6fcb1aae41703498f23f500f2bc" exitCode=0
Nov 03 11:50:58.582186 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.582187    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="668ab7bfabcc6fb3ae8e48e8226ef1adb43820641df68271b6fdc3f5b9b4dbda" exitCode=0
Nov 03 11:50:58.582186 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.582194    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="8597a652670ed492b99ef874be7c269e1b9ad5619a50707debd790a19a951360" exitCode=0
Nov 03 11:50:58.582186 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.582199    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="d85c0b7352c2700d65640f7479fbfcaeeb5a8606a5a3ebed25289dd5de730e3f" exitCode=0
Nov 03 11:50:58.582488 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.582206    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd63d182-450e-45aa-a997-4c98efe630dc" containerID="f9f3e7134df583d375db2dde2bf8a824f534167f81b1d0b88be59ba338995131" exitCode=0
Nov 03 11:50:58.609388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.609353    4546 generic.go:356] "Generic (PLEG): container finished" podID="5bcf5acf7ff1c4e9e6c3e7f2de3960f9" containerID="927fe6d6f023e6b06c3c0e6dd2b0d3cb08257c9761e99be8f4bf38b08c0fe72b" exitCode=0
Nov 03 11:50:58.613319 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.613300    4546 manager.go:324] Recovery completed
Nov 03 11:50:58.627320 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.627298    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.628427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.628409    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.628517 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.628440    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.628517 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.628453    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.629893 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.629869    4546 cpu_manager.go:226] "Starting" policy="none"
Nov 03 11:50:58.629893 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.629885    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.630020 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.629885    4546 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 11:50:58.630020 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.629978    4546 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 11:50:58.630113 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630095    4546 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 11:50:58.630143 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630105    4546 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 11:50:58.630143 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630120    4546 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 11:50:58.630143 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630125    4546 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 11:50:58.630143 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630131    4546 policy_none.go:50] "Start"
Nov 03 11:50:58.630143 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630139    4546 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 11:50:58.630283 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630148    4546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 11:50:58.630283 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630265    4546 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 11:50:58.630283 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630272    4546 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 11:50:58.630283 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630277    4546 policy_none.go:44] "Start"
Nov 03 11:50:58.630724 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630702    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.630818 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.630709    4546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:58.630818 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630739    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.630818 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.630784    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.635156 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.635131    4546 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 11:50:58.638805 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.638791    4546 manager.go:341] "Starting Device Plugin manager"
Nov 03 11:50:58.638874 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.638822    4546 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 11:50:58.638874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.638831    4546 server.go:87] "Starting device plugin registration server"
Nov 03 11:50:58.639145 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639133    4546 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 11:50:58.639175 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639148    4546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 11:50:58.639244 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639229    4546 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 11:50:58.639960 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639325    4546 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 11:50:58.639960 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639368    4546 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 11:50:58.639960 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639374    4546 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 11:50:58.640708 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.639431    4546 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T11:50:58.639356917Z","UUID":"4164bde4-8e7f-4122-8da8-1cff242cca88","Handler":null,"Name":"","Endpoint":""}
Nov 03 11:50:58.640884 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.640848    4546 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 11:50:58.640945 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.640904    4546 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-245.us-west-2.compute.internal\" not found"
Nov 03 11:50:58.641201 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.641176    4546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 11:50:58.641316 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.641240    4546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 11:50:58.666256 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.665733    4546 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"]
Nov 03 11:50:58.666372 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.666337    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.667214 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.667199    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.667307 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.667223    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.667307 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.667239    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.667482 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.667472    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.667654 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.667639    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.668252 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668233    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.668325 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668264    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.668325 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668273    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.668325 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668236    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.668453 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668336    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.668453 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668346    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.668453 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.668415    4546 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.668624 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668594    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5bcf5acf7ff1c4e9e6c3e7f2de3960f9","Type":"ContainerStarted","Data":"4cf3fa5e806f8516ce87c32a4c2a10d543472e423407aa8b352a8d78926fac65"}
Nov 03 11:50:58.668657 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668633    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5bcf5acf7ff1c4e9e6c3e7f2de3960f9","Type":"ContainerDied","Data":"927fe6d6f023e6b06c3c0e6dd2b0d3cb08257c9761e99be8f4bf38b08c0fe72b"}
Nov 03 11:50:58.668657 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668647    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5bcf5acf7ff1c4e9e6c3e7f2de3960f9","Type":"ContainerStarted","Data":"858d8382607a426d62055bd9721debd4cdaa2143e9b5382c17ae7f6bdc5d76b7"}
Nov 03 11:50:58.668712 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668658    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5cd06e439bdd68be11ac5140e2509e6a","Type":"ContainerStarted","Data":"a599be0a5a8491b2c12d062cf747779292f3c49a3ccdc181ee0b2316943931a9"}
Nov 03 11:50:58.668712 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668669    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal" event={"ID":"5cd06e439bdd68be11ac5140e2509e6a","Type":"ContainerStarted","Data":"1c19d749d537cc4770633bf101fc5040be4e4e0825c5b48f2a425ffd5d473473"}
Nov 03 11:50:58.668712 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.668692    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.669383 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.669368    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.669447 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.669396    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.669447 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.669406    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.669539 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.669525    4546 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-245.us-west-2.compute.internal\" not found" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.713164 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.713128    4546 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:58.713164 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.713130    4546 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:58.713332 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.713317    4546 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 11:50:58.739602 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.739550    4546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:58.740522 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.740501    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:58.740602 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.740539    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:58.740602 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.740554    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:58.740602 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.740591    4546 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.760403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.760381    4546 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:58.801866 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.801836    4546 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:58.813849 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.813820    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.813849 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.813849    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.814001 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.813868    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cd06e439bdd68be11ac5140e2509e6a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5cd06e439bdd68be11ac5140e2509e6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.830570 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.830547    4546 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.850040 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.850017    4546 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.850111 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.850076    4546 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.850624 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.850608    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeNotReady"
Nov 03 11:50:58.850699 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.850641    4546 setters.go:546] "Node became not ready" node="ip-10-0-136-245.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T11:50:58Z","lastTransitionTime":"2026-11-03T11:50:58Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 11:50:58.866320 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.866296    4546 nodeinfomanager.go:471] existing CSINode "ip-10-0-136-245.us-west-2.compute.internal" is owned by different node (oldNodeID="09790f07-d972-4db1-bfaf-d32bf3854d22", newNodeID=""), cleaning up...
Nov 03 11:50:58.887097 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.887079    4546 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:58.887159 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.887117    4546 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.887159 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.887129    4546 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.914308 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.914280    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.914418 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.914318    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.914418 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.914339    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cd06e439bdd68be11ac5140e2509e6a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5cd06e439bdd68be11ac5140e2509e6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.914418 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.914387    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.914509 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.914412    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcf5acf7ff1c4e9e6c3e7f2de3960f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5bcf5acf7ff1c4e9e6c3e7f2de3960f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.914509 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.914389    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cd06e439bdd68be11ac5140e2509e6a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" (UID: \"5cd06e439bdd68be11ac5140e2509e6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:58.942404 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:58.942375    4546 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:58.942507 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:58.942426    4546 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:59.516687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.516659    4546 apiserver.go:52] "Watching apiserver"
Nov 03 11:50:59.612711 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.612682    4546 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 11:50:59.613196 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.613178    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z","openshift-image-registry/node-ca-vkpbq","openshift-ingress-canary/ingress-canary-22bn2","openshift-multus/multus-additional-cni-plugins-rq7tf","kube-system/konnectivity-agent-wh554","openshift-cluster-node-tuning-operator/tuned-j62p5","openshift-insights/periodic-gathering-gs7fz-8qhqm","openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm","openshift-ovn-kubernetes/ovnkube-node-sqqxm","openshift-network-operator/iptables-alerter-hv5fv","openshift-dns/node-resolver-ssfqd","openshift-insights/insights-runtime-extractor-4c659","openshift-service-ca/service-ca-5fffc97bdf-4st2x","kube-system/global-pull-secret-syncer-lpbww","openshift-dns/dns-default-c9tgf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal","openshift-multus/multus-2f4bq","openshift-multus/network-metrics-daemon-s79nv","openshift-network-diagnostics/network-check-target-pgfhl"]
Nov 03 11:50:59.614340 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.614319    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:59.617499 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.617482    4546 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:59.617696 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.617680    4546 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:59.618154 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.618140    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.622541 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.622281    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.629450 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.629431    4546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 11:50:59.663382 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.663358    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.665696 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665669    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665717    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665717    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665717    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665722    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665776    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665816    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665830    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665839    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.665835 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665839    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665861    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665897    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.665977    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666010    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666028    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666048    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666077    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xf9nj\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666092    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666110    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666117    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666150    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666168    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-65pss\""
Nov 03 11:50:59.666277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666202    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666370    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666379    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-swh4m\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666385    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666388    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666397    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jp4k\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666433    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666460    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666538    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6gwtx\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666539    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jp68j\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666554    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666561    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666575    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666578    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666616    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666657    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxdwl\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666710    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g5g7l\""
Nov 03 11:50:59.666812 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.666777    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vn7bf\""
Nov 03 11:50:59.673998 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.673984    4546 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:59.674052 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:59.674024    4546 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:59.674264 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.674249    4546 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:59.674309 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:59.674281    4546 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-245.us-west-2.compute.internal"
Nov 03 11:50:59.703811 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703779    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 11:50:59.703965 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703841    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzgk2\""
Nov 03 11:50:59.703965 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703894    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 11:50:59.703965 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703926    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nxjhp\""
Nov 03 11:50:59.703965 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703946    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:50:59.704113 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703984    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:50:59.704113 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.703986    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pf74h\""
Nov 03 11:50:59.706080 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.706063    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7tbzl\""
Nov 03 11:50:59.706144 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.706111    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 11:50:59.706192 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.706172    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7npm\""
Nov 03 11:50:59.706192 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.706175    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 11:50:59.706320 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.706308    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fw7f8\""
Nov 03 11:50:59.718739 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718698    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-cert\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:59.718819 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718743    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.718887 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718872    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-socket-dir-parent\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.718927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-conf-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.718927 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718919    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-log-socket\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.718996 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718937    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.718996 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.718966    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78648619-04c1-4830-ba9a-9b9f8f568550-data-volume\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.719066 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719000    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-system-cni-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.719066 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719025    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-cert\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:59.719066 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719036    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78648619-04c1-4830-ba9a-9b9f8f568550-data-volume\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719065    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-cni-binary-copy\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719110    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-slash\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719130    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719178    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719275    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719295    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719347    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719364    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-dbus\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.719403 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719385    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-node-log\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719430    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45488046-69c7-4931-b7b7-6efe7be609df-ovn-node-metrics-cert\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719447    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-script-lib\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719462    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719476    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9tj\" (UniqueName: \"kubernetes.io/projected/372ffe3a-8e0e-4e2e-ab32-074098ec0133-kube-api-access-fg9tj\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719491    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719528    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-cni-binary-copy\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719698    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-kubelet\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719730    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntspk\" (UniqueName: \"kubernetes.io/projected/e5e9caab-b867-45ec-b81c-b74d1821302d-kube-api-access-ntspk\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719765    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-kubelet-config\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.719796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719780    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-script-lib\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719809    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcjz\" (UniqueName: \"kubernetes.io/projected/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kube-api-access-7rcjz\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719928    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-conf\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.719975    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45488046-69c7-4931-b7b7-6efe7be609df-ovn-node-metrics-cert\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720005    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-var-lib-kubelet\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720028    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-hostroot\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720048    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgk6\" (UniqueName: \"kubernetes.io/projected/6a115576-c68b-4963-a7bc-5cc5232602e6-kube-api-access-jwgk6\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720075    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-kubelet\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720113    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-systemd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720165 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720151    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720193    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd2k\" (UniqueName: \"kubernetes.io/projected/78648619-04c1-4830-ba9a-9b9f8f568550-kube-api-access-wxd2k\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720219    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-agent-certs\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720244    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-ovn\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720263    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720308    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z49h\" (UniqueName: \"kubernetes.io/projected/dd63d182-450e-45aa-a997-4c98efe630dc-kube-api-access-4z49h\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720323    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-tuned\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720380    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-netns\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720406    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-netd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720431    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-tuned\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720437    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720432    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57x2n\" (UniqueName: \"kubernetes.io/projected/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-kube-api-access-57x2n\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720484    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-host-slash\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720504    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv992\" (UniqueName: \"kubernetes.io/projected/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-kube-api-access-mv992\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720616    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6l8\" (UniqueName: \"kubernetes.io/projected/870c96d8-257f-439b-a780-6a4f53964f50-kube-api-access-cf6l8\") pod \"migrator-66df5d496b-wspzm\" (UID: \"870c96d8-257f-439b-a780-6a4f53964f50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:59.720645 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720650    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-agent-certs\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720654    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysconfig\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720695    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720731    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-systemd\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720791    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-cabundle\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720824    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-device-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720849    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-serviceca\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720893    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/372ffe3a-8e0e-4e2e-ab32-074098ec0133-config-volume\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720926    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.720954    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-sys\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721087    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-system-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721117    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-cnibin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721146    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-bin\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721166    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721176    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-iptables-alerter-script\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721193    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-host\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721216    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372ffe3a-8e0e-4e2e-ab32-074098ec0133-tmp-dir\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721292    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-kubernetes\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721319    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372ffe3a-8e0e-4e2e-ab32-074098ec0133-tmp-dir\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.721349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721326    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-daemon-config\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721364    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721393    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78648619-04c1-4830-ba9a-9b9f8f568550-crio-socket\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721421    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721422    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-iptables-alerter-script\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721458    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721485    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-serviceca\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721490    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721518    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-daemon-config\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721535    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7lm\" (UniqueName: \"kubernetes.io/projected/fdbf6c89-33f3-48a2-aa96-16a26e336e34-kube-api-access-hf7lm\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721616    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-k8s-cni-cncf-io\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721646    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721704    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-bin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721721    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721766    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2znb\" (UniqueName: \"kubernetes.io/projected/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-kube-api-access-l2znb\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721836    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721869    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721887    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-lib-modules\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721920    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-etc-kubernetes\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721947    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-env-overrides\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.721963    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075924a-500c-4f50-ab6c-dbb51839181c-tmp-dir\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722007    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpccs\" (UniqueName: \"kubernetes.io/projected/b075924a-500c-4f50-ab6c-dbb51839181c-kube-api-access-gpccs\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722032    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-modprobe-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722046    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075924a-500c-4f50-ab6c-dbb51839181c-tmp-dir\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722053    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-tmp\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722099    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd63d182-450e-45aa-a997-4c98efe630dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722117    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-netns\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722105    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbf6c89-33f3-48a2-aa96-16a26e336e34-tmp\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722130    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-env-overrides\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722139    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-config\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722178    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-key\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722234    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k9m\" (UniqueName: \"kubernetes.io/projected/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-kube-api-access-t5k9m\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.722262 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722262    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722294    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-konnectivity-ca\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722321    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-run\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722327    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45488046-69c7-4931-b7b7-6efe7be609df-ovnkube-config\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722352    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-multus\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722377    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-multus-certs\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722407    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-key\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722419    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722446    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56tkk\" (UniqueName: \"kubernetes.io/projected/45488046-69c7-4931-b7b7-6efe7be609df-kube-api-access-56tkk\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722471    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b075924a-500c-4f50-ab6c-dbb51839181c-hosts-file\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722474    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ead3a7f-804d-48a7-9dae-ec5df206cbbe-konnectivity-ca\") pod \"konnectivity-agent-wh554\" (UID: \"2ead3a7f-804d-48a7-9dae-ec5df206cbbe\") " pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722494    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722526    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-cnibin\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722560    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-host\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722584    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-systemd-units\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722617    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-var-lib-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722644    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-etc-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.723312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.722657    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/606502b8-aa74-4661-94fe-43d62bfb8d40-original-pull-secret\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.765149 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765118    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.765149 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765137    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.765149 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765123    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765153    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765191    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765153    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765155    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765209    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765220    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765222    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765215    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 11:50:59.765353 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.765203    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 11:50:59.766166 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.766153    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 11:50:59.771053 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.771001    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78648619-04c1-4830-ba9a-9b9f8f568550-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.771154 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.771109    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/372ffe3a-8e0e-4e2e-ab32-074098ec0133-config-volume\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.771207 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.771196    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6a115576-c68b-4963-a7bc-5cc5232602e6-signing-cabundle\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:59.771829 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.771807    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntspk\" (UniqueName: \"kubernetes.io/projected/e5e9caab-b867-45ec-b81c-b74d1821302d-kube-api-access-ntspk\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.771946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.771815    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknkk\" (UniqueName: \"kubernetes.io/projected/62d43ee3-52a9-4c5f-80f1-81b795f04ea6-kube-api-access-tknkk\") pod \"network-check-target-pgfhl\" (UID: \"62d43ee3-52a9-4c5f-80f1-81b795f04ea6\") " pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:50:59.771946 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.771928    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.772137 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772109    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcjz\" (UniqueName: \"kubernetes.io/projected/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kube-api-access-7rcjz\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.772243 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772164    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.772243 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772183    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd2k\" (UniqueName: \"kubernetes.io/projected/78648619-04c1-4830-ba9a-9b9f8f568550-kube-api-access-wxd2k\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.772243 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772214    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") pod \"periodic-gathering-gs7fz-8qhqm\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") " pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:50:59.772411 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772265    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9tj\" (UniqueName: \"kubernetes.io/projected/372ffe3a-8e0e-4e2e-ab32-074098ec0133-kube-api-access-fg9tj\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.772461 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772408    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgk6\" (UniqueName: \"kubernetes.io/projected/6a115576-c68b-4963-a7bc-5cc5232602e6-kube-api-access-jwgk6\") pod \"service-ca-5fffc97bdf-4st2x\" (UID: \"6a115576-c68b-4963-a7bc-5cc5232602e6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4st2x"
Nov 03 11:50:59.772635 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772614    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z49h\" (UniqueName: \"kubernetes.io/projected/dd63d182-450e-45aa-a997-4c98efe630dc-kube-api-access-4z49h\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.772865 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.772850    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6l8\" (UniqueName: \"kubernetes.io/projected/870c96d8-257f-439b-a780-6a4f53964f50-kube-api-access-cf6l8\") pod \"migrator-66df5d496b-wspzm\" (UID: \"870c96d8-257f-439b-a780-6a4f53964f50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wspzm"
Nov 03 11:50:59.773022 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.773003    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv992\" (UniqueName: \"kubernetes.io/projected/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-kube-api-access-mv992\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.773170 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.773157    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57x2n\" (UniqueName: \"kubernetes.io/projected/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-kube-api-access-57x2n\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:59.773536 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.773521    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpccs\" (UniqueName: \"kubernetes.io/projected/b075924a-500c-4f50-ab6c-dbb51839181c-kube-api-access-gpccs\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.773922 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.773903    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56tkk\" (UniqueName: \"kubernetes.io/projected/45488046-69c7-4931-b7b7-6efe7be609df-kube-api-access-56tkk\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.774094 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.774079    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2znb\" (UniqueName: \"kubernetes.io/projected/88b8fe32-ab27-46d3-bc8d-e562b27a82cf-kube-api-access-l2znb\") pod \"ingress-canary-22bn2\" (UID: \"88b8fe32-ab27-46d3-bc8d-e562b27a82cf\") " pod="openshift-ingress-canary/ingress-canary-22bn2"
Nov 03 11:50:59.774767 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.774733    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k9m\" (UniqueName: \"kubernetes.io/projected/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-kube-api-access-t5k9m\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.774830 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.774814    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7lm\" (UniqueName: \"kubernetes.io/projected/fdbf6c89-33f3-48a2-aa96-16a26e336e34-kube-api-access-hf7lm\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.823410 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823375    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-slash\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.823410 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823411    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823434    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823453    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823476    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823481    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-slash\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823500    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-dbus\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823524    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-node-log\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823529    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823545    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823542    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:59.823552    4546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823583    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-kubelet\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823606    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-kubelet-config\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823623    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-conf\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.823626 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823622    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-dbus\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823625    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-kubelet\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823583    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-node-log\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823660    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/606502b8-aa74-4661-94fe-43d62bfb8d40-kubelet-config\") pod \"global-pull-secret-syncer-lpbww\" (UID: \"606502b8-aa74-4661-94fe-43d62bfb8d40\") " pod="kube-system/global-pull-secret-syncer-lpbww"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:59.823664    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:51:00.323640506 +0000 UTC m=+1.895224426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : secret "metrics-daemon-secret" not found
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823691    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-var-lib-kubelet\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823709    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-hostroot\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823732    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-kubelet\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823605    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823736    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-conf\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823783    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-var-lib-kubelet\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823782    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-systemd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823804    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-hostroot\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823812    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823805    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-kubelet\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823846    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-ovn\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823853    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823847    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-systemd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823870    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-netns\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823872    4546 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823885    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-netd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823900    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-host-slash\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823917    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysconfig\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823934    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823928    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-netns\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823959    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-systemd\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823964    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b8775a-e0e0-4cf6-9be5-9df4891d6ccc-host-slash\") pod \"iptables-alerter-hv5fv\" (UID: \"87b8775a-e0e0-4cf6-9be5-9df4891d6ccc\") " pod="openshift-network-operator/iptables-alerter-hv5fv"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823981    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysconfig\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824004    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-netd\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824029    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-device-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824059    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-sys\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824011    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-systemd\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824064    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-device-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.823879    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-ovn\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824084    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-system-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824086    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-sysctl-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824103    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-sys\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824118    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-cnibin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824167    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-system-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824199    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-bin\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824225    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-host\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824224    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-cnibin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824248    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-kubernetes\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824256    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-cni-bin\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.824231 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824267    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67983f1-d777-4fde-a01b-24d8a2a4f3c5-host\") pod \"node-ca-vkpbq\" (UID: \"c67983f1-d777-4fde-a01b-24d8a2a4f3c5\") " pod="openshift-image-registry/node-ca-vkpbq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824268    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824289    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-kubernetes\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824291    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78648619-04c1-4830-ba9a-9b9f8f568550-crio-socket\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824306    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-run-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824316    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824340    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-k8s-cni-cncf-io\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824347    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78648619-04c1-4830-ba9a-9b9f8f568550-crio-socket\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824366    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-bin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824394    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824401    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-k8s-cni-cncf-io\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824407    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-bin\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824421    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-lib-modules\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824444    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-etc-kubernetes\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824448    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824400    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824469    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-modprobe-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824494    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-netns\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824499    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-etc-kubernetes\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824520    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824523    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-lib-modules\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824540    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-host-run-netns\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824543    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-run\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824567    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-etc-modprobe-d\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824566    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-multus\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:59.824586    4546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824596    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-multus-certs\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824606    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-var-lib-cni-multus\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824621    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b075924a-500c-4f50-ab6c-dbb51839181c-hosts-file\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:50:59.824642    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:00.324626416 +0000 UTC m=+1.896210337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824647    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-host-run-multus-certs\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824639    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-run\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824661    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b075924a-500c-4f50-ab6c-dbb51839181c-hosts-file\") pod \"node-resolver-ssfqd\" (UID: \"b075924a-500c-4f50-ab6c-dbb51839181c\") " pod="openshift-dns/node-resolver-ssfqd"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824666    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824694    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824716    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-cnibin\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824765    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-host\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824788    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-cnibin\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824789    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-systemd-units\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824824    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-var-lib-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824826    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-systemd-units\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824826    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbf6c89-33f3-48a2-aa96-16a26e336e34-host\") pod \"tuned-j62p5\" (UID: \"fdbf6c89-33f3-48a2-aa96-16a26e336e34\") " pod="openshift-cluster-node-tuning-operator/tuned-j62p5"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824845    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-var-lib-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824845    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-etc-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824882    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824897    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-socket-dir-parent\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824881    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-etc-openvswitch\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824920    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-cni-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824912    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-conf-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824957    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-log-socket\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824959    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-socket-dir-parent\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824934    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e9caab-b867-45ec-b81c-b74d1821302d-multus-conf-dir\") pod \"multus-2f4bq\" (UID: \"e5e9caab-b867-45ec-b81c-b74d1821302d\") " pod="openshift-multus/multus-2f4bq"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.824982    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.825006    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45488046-69c7-4931-b7b7-6efe7be609df-log-socket\") pod \"ovnkube-node-sqqxm\" (UID: \"45488046-69c7-4931-b7b7-6efe7be609df\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.825008    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-system-cni-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.825024    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bee7ca-d883-4a3b-94c7-9ad031cad22f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2m8z\" (UID: \"a9bee7ca-d883-4a3b-94c7-9ad031cad22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2m8z"
Nov 03 11:50:59.825688 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.825035    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd63d182-450e-45aa-a997-4c98efe630dc-system-cni-dir\") pod \"multus-additional-cni-plugins-rq7tf\" (UID: \"dd63d182-450e-45aa-a997-4c98efe630dc\") " pod="openshift-multus/multus-additional-cni-plugins-rq7tf"
Nov 03 11:50:59.827158 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.826296    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78648619-04c1-4830-ba9a-9b9f8f568550-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c659\" (UID: \"78648619-04c1-4830-ba9a-9b9f8f568550\") " pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.922788 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.922743    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4c659"
Nov 03 11:50:59.957309 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:50:59.957285    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vkpbq_c67983f1-d777-4fde-a01b-24d8a2a4f3c5/node-ca/0.log"
Nov 03 11:51:00.328338 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.328289    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:51:00.328614 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.328368    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:00.328614 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:00.328440    4546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:51:00.328614 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:00.328457    4546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:51:00.328614 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:00.328501    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:51:01.328487665 +0000 UTC m=+2.900071579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : secret "metrics-daemon-secret" not found
Nov 03 11:51:00.328614 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:00.328515    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:01.328509398 +0000 UTC m=+2.900093288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:51:00.358298 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.358270    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-22bn2_88b8fe32-ab27-46d3-bc8d-e562b27a82cf/serve-healthcheck-canary/0.log"
Nov 03 11:51:00.363838 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.363807    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4c659"]
Nov 03 11:51:00.367981 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:00.367952    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78648619_04c1_4830_ba9a_9b9f8f568550.slice/crio-619d3ccd2b1cd77e49a3887cc9843ccb2420a4696beec02ac61068e3650a1929 WatchSource:0}: Error finding container 619d3ccd2b1cd77e49a3887cc9843ccb2420a4696beec02ac61068e3650a1929: Status 404 returned error can't find the container with id 619d3ccd2b1cd77e49a3887cc9843ccb2420a4696beec02ac61068e3650a1929
Nov 03 11:51:00.447899 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.447875    4546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:51:00.625966 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.625872    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c659" event={"ID":"78648619-04c1-4830-ba9a-9b9f8f568550","Type":"ContainerStarted","Data":"c967f8c688fc9015d888423f1bae771745768805bf878c2d0cb926f3645444c8"}
Nov 03 11:51:00.625966 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.625950    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c659" event={"ID":"78648619-04c1-4830-ba9a-9b9f8f568550","Type":"ContainerStarted","Data":"619d3ccd2b1cd77e49a3887cc9843ccb2420a4696beec02ac61068e3650a1929"}
Nov 03 11:51:00.958949 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:00.958877    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wspzm_870c96d8-257f-439b-a780-6a4f53964f50/migrator/0.log"
Nov 03 11:51:01.336233 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:01.336195    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:01.336233 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:01.336236    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:51:01.336478 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:01.336331    4546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:51:01.336478 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:01.336337    4546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 11:51:01.336478 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:01.336393    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:51:03.336381404 +0000 UTC m=+4.907965295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : secret "metrics-daemon-secret" not found
Nov 03 11:51:01.336478 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:01.336406    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls podName:372ffe3a-8e0e-4e2e-ab32-074098ec0133 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:03.336400041 +0000 UTC m=+4.907983931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls") pod "dns-default-c9tgf" (UID: "372ffe3a-8e0e-4e2e-ab32-074098ec0133") : secret "dns-default-metrics-tls" not found
Nov 03 11:51:01.405986 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:01.405952    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:51:02.635709 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:02.635678    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c659" event={"ID":"78648619-04c1-4830-ba9a-9b9f8f568550","Type":"ContainerStarted","Data":"d40c70f719742a3a13cd52da5c67ad1cbf46365b393a5ba1f1504f0eb2af27ba"}
Nov 03 11:51:03.348907 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:03.348853    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:51:03.349094 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:03.348976    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:03.349094 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:03.349018    4546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 11:51:03.349200 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:03.349103    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs podName:c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce nodeName:}" failed. No retries permitted until 2026-11-03 11:51:07.349082881 +0000 UTC m=+8.920666786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs") pod "network-metrics-daemon-s79nv" (UID: "c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce") : secret "metrics-daemon-secret" not found
Nov 03 11:51:03.351859 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:03.351834    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/372ffe3a-8e0e-4e2e-ab32-074098ec0133-metrics-tls\") pod \"dns-default-c9tgf\" (UID: \"372ffe3a-8e0e-4e2e-ab32-074098ec0133\") " pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:03.489021 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:03.488991    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:51:03.519317 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:03.519285    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:03.878110 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:03.878075    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c9tgf"]
Nov 03 11:51:03.881934 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:03.881906    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372ffe3a_8e0e_4e2e_ab32_074098ec0133.slice/crio-2c82fc9402640ca312b0d28bf9c496fcaeb91f3e069f9768b3859798e169b964 WatchSource:0}: Error finding container 2c82fc9402640ca312b0d28bf9c496fcaeb91f3e069f9768b3859798e169b964: Status 404 returned error can't find the container with id 2c82fc9402640ca312b0d28bf9c496fcaeb91f3e069f9768b3859798e169b964
Nov 03 11:51:04.647044 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:04.647012    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c659" event={"ID":"78648619-04c1-4830-ba9a-9b9f8f568550","Type":"ContainerStarted","Data":"fcc3e80d80683bfa2901babbb2c2cc3edcddf3a1c8c015b9bd9c293c98763d74"}
Nov 03 11:51:04.648264 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:04.648241    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9tgf" event={"ID":"372ffe3a-8e0e-4e2e-ab32-074098ec0133","Type":"ContainerStarted","Data":"2c82fc9402640ca312b0d28bf9c496fcaeb91f3e069f9768b3859798e169b964"}
Nov 03 11:51:07.374100 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:07.374058    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:51:07.376596 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:07.376571    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce-metrics-certs\") pod \"network-metrics-daemon-s79nv\" (UID: \"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce\") " pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:51:07.415580 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:07.415538    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s79nv"
Nov 03 11:51:07.650177 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:07.650101    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:51:07.772521 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:07.772428    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s79nv"]
Nov 03 11:51:07.776652 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:07.776624    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c62f30_a1c3_40dc_b5cf_53bfa19fd9ce.slice/crio-c137fa34744358cdc58b6746e15910433ced4a522b59d91224988848a04783d2 WatchSource:0}: Error finding container c137fa34744358cdc58b6746e15910433ced4a522b59d91224988848a04783d2: Status 404 returned error can't find the container with id c137fa34744358cdc58b6746e15910433ced4a522b59d91224988848a04783d2
Nov 03 11:51:08.310499 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:08.310470    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:51:08.310707 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:08.310693    4546 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 11:51:08.662190 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:08.662080    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s79nv" event={"ID":"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce","Type":"ContainerStarted","Data":"c137fa34744358cdc58b6746e15910433ced4a522b59d91224988848a04783d2"}
Nov 03 11:51:09.164918 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:09.164879    4546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-245.us-west-2.compute.internal" event="NodeReady"
Nov 03 11:51:10.670897 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:10.670853    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s79nv" event={"ID":"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce","Type":"ContainerStarted","Data":"44336a9ba416dc5cce7c8015c37de98fb93bab8769a35917badfebda00836466"}
Nov 03 11:51:10.671307 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:10.670922    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s79nv" event={"ID":"c0c62f30-a1c3-40dc-b5cf-53bfa19fd9ce","Type":"ContainerStarted","Data":"1c30edca3f2121e46a40857b1b2a6680cbfab783ee7420cf9bc68d85d2215fd1"}
Nov 03 11:51:11.675184 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:11.675150    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9tgf" event={"ID":"372ffe3a-8e0e-4e2e-ab32-074098ec0133","Type":"ContainerStarted","Data":"ed15b7e71420ff701caef0234ac5389c5099b75f00a763b7129b2e3a99053a92"}
Nov 03 11:51:11.675551 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:11.675198    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9tgf" event={"ID":"372ffe3a-8e0e-4e2e-ab32-074098ec0133","Type":"ContainerStarted","Data":"69f748687fbfd17d66ff251a477d45386f9b66f54f175ade99bc8dd9cbe6a35c"}
Nov 03 11:51:12.177248 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.177217    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-sbqz8"]
Nov 03 11:51:12.180607 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.180590    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.208909 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.208872    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75caa47e-e5fd-4bf0-96fb-ddcf53ea3745-tmp\") pod \"downloads-6dd789c8b7-sbqz8\" (UID: \"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745\") " pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.209042 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.208935    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkdh\" (UniqueName: \"kubernetes.io/projected/75caa47e-e5fd-4bf0-96fb-ddcf53ea3745-kube-api-access-rzkdh\") pod \"downloads-6dd789c8b7-sbqz8\" (UID: \"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745\") " pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.228893 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.228870    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 11:51:12.231652 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.231634    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-rkghw\""
Nov 03 11:51:12.231839 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.231826    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 11:51:12.267168 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.267124    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b5cc5f87-cxg2p"]
Nov 03 11:51:12.270263 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.270244    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310206 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310182    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28f8\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-kube-api-access-j28f8\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310352 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310225    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75caa47e-e5fd-4bf0-96fb-ddcf53ea3745-tmp\") pod \"downloads-6dd789c8b7-sbqz8\" (UID: \"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745\") " pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.310352 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310250    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7897b960-e8d6-4917-906f-b7d87b648bb8-installation-pull-secrets\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310429 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310382    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7897b960-e8d6-4917-906f-b7d87b648bb8-registry-certificates\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310429 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310406    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7897b960-e8d6-4917-906f-b7d87b648bb8-ca-trust-extracted\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310429 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310422    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-bound-sa-token\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310524 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310473    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7897b960-e8d6-4917-906f-b7d87b648bb8-image-registry-private-configuration\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310524 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310511    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7897b960-e8d6-4917-906f-b7d87b648bb8-trusted-ca\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310580 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310551    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdh\" (UniqueName: \"kubernetes.io/projected/75caa47e-e5fd-4bf0-96fb-ddcf53ea3745-kube-api-access-rzkdh\") pod \"downloads-6dd789c8b7-sbqz8\" (UID: \"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745\") " pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.310612 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310584    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-registry-tls\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.310612 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.310586    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75caa47e-e5fd-4bf0-96fb-ddcf53ea3745-tmp\") pod \"downloads-6dd789c8b7-sbqz8\" (UID: \"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745\") " pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.373384 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.373353    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 11:51:12.373554 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.373464    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tvsxn\""
Nov 03 11:51:12.373554 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.373466    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 11:51:12.373554 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.373497    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 11:51:12.373554 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.373540    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 11:51:12.373811 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.373793    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b5cc5f87-cxg2p"]
Nov 03 11:51:12.375609 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.375594    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sbqz8"]
Nov 03 11:51:12.377500 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.377483    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkdh\" (UniqueName: \"kubernetes.io/projected/75caa47e-e5fd-4bf0-96fb-ddcf53ea3745-kube-api-access-rzkdh\") pod \"downloads-6dd789c8b7-sbqz8\" (UID: \"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745\") " pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.411862 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.411832    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28f8\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-kube-api-access-j28f8\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.411992 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.411875    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7897b960-e8d6-4917-906f-b7d87b648bb8-installation-pull-secrets\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.411992 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.411893    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7897b960-e8d6-4917-906f-b7d87b648bb8-registry-certificates\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412070 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412016    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7897b960-e8d6-4917-906f-b7d87b648bb8-ca-trust-extracted\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412070 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412045    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-bound-sa-token\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412157 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412075    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7897b960-e8d6-4917-906f-b7d87b648bb8-image-registry-private-configuration\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412240 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412220    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7897b960-e8d6-4917-906f-b7d87b648bb8-trusted-ca\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412354 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412281    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-registry-tls\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412442 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412414    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7897b960-e8d6-4917-906f-b7d87b648bb8-ca-trust-extracted\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.412811 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.412790    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7897b960-e8d6-4917-906f-b7d87b648bb8-registry-certificates\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.413266 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.413204    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7897b960-e8d6-4917-906f-b7d87b648bb8-trusted-ca\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.414586 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.414568    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7897b960-e8d6-4917-906f-b7d87b648bb8-image-registry-private-configuration\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.414586 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.414580    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7897b960-e8d6-4917-906f-b7d87b648bb8-installation-pull-secrets\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.414893 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.414878    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-registry-tls\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.464472 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.464449    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-bound-sa-token\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.464596 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.464507    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28f8\" (UniqueName: \"kubernetes.io/projected/7897b960-e8d6-4917-906f-b7d87b648bb8-kube-api-access-j28f8\") pod \"image-registry-b5cc5f87-cxg2p\" (UID: \"7897b960-e8d6-4917-906f-b7d87b648bb8\") " pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.492420 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.492395    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:12.584338 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.584310    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:12.678651 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.678621    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:12.897034 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.896960    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sbqz8"]
Nov 03 11:51:12.900647 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:12.900598    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75caa47e_e5fd_4bf0_96fb_ddcf53ea3745.slice/crio-28402694448a7d8d75dd312be8e2a9e0f1317fbe0004e9157f2058350aa8d829 WatchSource:0}: Error finding container 28402694448a7d8d75dd312be8e2a9e0f1317fbe0004e9157f2058350aa8d829: Status 404 returned error can't find the container with id 28402694448a7d8d75dd312be8e2a9e0f1317fbe0004e9157f2058350aa8d829
Nov 03 11:51:12.946527 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:12.946498    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b5cc5f87-cxg2p"]
Nov 03 11:51:12.949389 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:12.949367    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7897b960_e8d6_4917_906f_b7d87b648bb8.slice/crio-89a598ed1721b7276cce792c19dd736334a8b631cae5dbf20c32eea5ace4c7d7 WatchSource:0}: Error finding container 89a598ed1721b7276cce792c19dd736334a8b631cae5dbf20c32eea5ace4c7d7: Status 404 returned error can't find the container with id 89a598ed1721b7276cce792c19dd736334a8b631cae5dbf20c32eea5ace4c7d7
Nov 03 11:51:13.032625 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.032600    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"]
Nov 03 11:51:13.035675 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.035658    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:13.084379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.084348    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 11:51:13.084539 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.084421    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fkwpw\""
Nov 03 11:51:13.090333 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.090301    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"]
Nov 03 11:51:13.118192 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.118160    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac1369cc-8144-46c9-a267-c4586d82f99e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lztc9\" (UID: \"ac1369cc-8144-46c9-a267-c4586d82f99e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:13.218784 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.218722    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac1369cc-8144-46c9-a267-c4586d82f99e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lztc9\" (UID: \"ac1369cc-8144-46c9-a267-c4586d82f99e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:13.221254 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.221233    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac1369cc-8144-46c9-a267-c4586d82f99e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lztc9\" (UID: \"ac1369cc-8144-46c9-a267-c4586d82f99e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:13.347364 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.347334    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:13.683693 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.683618    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sbqz8" event={"ID":"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745","Type":"ContainerStarted","Data":"28402694448a7d8d75dd312be8e2a9e0f1317fbe0004e9157f2058350aa8d829"}
Nov 03 11:51:13.685152 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.685121    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p" event={"ID":"7897b960-e8d6-4917-906f-b7d87b648bb8","Type":"ContainerStarted","Data":"47fac3aa0e135ecb36cdbaa846abc92210454a13fe855897be80f744e1bec13d"}
Nov 03 11:51:13.685253 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.685164    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p" event={"ID":"7897b960-e8d6-4917-906f-b7d87b648bb8","Type":"ContainerStarted","Data":"89a598ed1721b7276cce792c19dd736334a8b631cae5dbf20c32eea5ace4c7d7"}
Nov 03 11:51:13.708926 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.708897    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"]
Nov 03 11:51:13.711788 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:13.711766    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1369cc_8144_46c9_a267_c4586d82f99e.slice/crio-762b51574c8039f8d8f0a33686b330dca6a39995ddf1550c975b11b1f268119d WatchSource:0}: Error finding container 762b51574c8039f8d8f0a33686b330dca6a39995ddf1550c975b11b1f268119d: Status 404 returned error can't find the container with id 762b51574c8039f8d8f0a33686b330dca6a39995ddf1550c975b11b1f268119d
Nov 03 11:51:13.795381 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:13.795321    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p" podStartSLOduration=1.7953084910000001 podStartE2EDuration="1.795308491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:13.794475733 +0000 UTC m=+15.366059665" watchObservedRunningTime="2026-11-03 11:51:13.795308491 +0000 UTC m=+15.366892403"
Nov 03 11:51:14.548232 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:14.548196    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wh554"
Nov 03 11:51:14.689872 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:14.689834    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9" event={"ID":"ac1369cc-8144-46c9-a267-c4586d82f99e","Type":"ContainerStarted","Data":"762b51574c8039f8d8f0a33686b330dca6a39995ddf1550c975b11b1f268119d"}
Nov 03 11:51:14.690314 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:14.690084    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:15.694910 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:15.694867    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9" event={"ID":"ac1369cc-8144-46c9-a267-c4586d82f99e","Type":"ContainerStarted","Data":"a8afefabf63dea9935724f1bbca9ac4285947c7f345a6bfec7b290368d1c70c0"}
Nov 03 11:51:15.695397 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:15.695155    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:15.699615 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:15.699596    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9"
Nov 03 11:51:15.801163 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:15.801071    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lztc9" podStartSLOduration=2.033724187 podStartE2EDuration="3.801057217s" totalImagesPullingTime="1.76733303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:15.800506263 +0000 UTC m=+17.372090193" watchObservedRunningTime="2026-11-03 11:51:15.801057217 +0000 UTC m=+17.372641126"
Nov 03 11:51:22.536090 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:22.536060    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqqxm"
Nov 03 11:51:22.687882 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:22.687855    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c9tgf"
Nov 03 11:51:23.048609 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.048573    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mpbk9"]
Nov 03 11:51:23.073287 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.073261    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093385 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093359    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mc9\" (UniqueName: \"kubernetes.io/projected/17f1df83-bba7-4eae-8a25-bfb807a364c5-kube-api-access-l7mc9\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093523 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093402    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-wtmp\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093523 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093427    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093614 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093535    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f1df83-bba7-4eae-8a25-bfb807a364c5-metrics-client-ca\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093614 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093568    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-root\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093614 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093605    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093780 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093710    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-textfile\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093780 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093738    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-sys\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.093889 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.093782    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-tls\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.192478 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.192450    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 11:51:23.192478 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.192449    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 11:51:23.192699 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.192653    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 11:51:23.192699 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.192667    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 11:51:23.193163 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.193145    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s8zh8\""
Nov 03 11:51:23.193235 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.193146    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 11:51:23.193269 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.193239    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 11:51:23.194317 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194291    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mc9\" (UniqueName: \"kubernetes.io/projected/17f1df83-bba7-4eae-8a25-bfb807a364c5-kube-api-access-l7mc9\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194409 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194330    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-wtmp\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194409 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194362    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194409 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194392    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f1df83-bba7-4eae-8a25-bfb807a364c5-metrics-client-ca\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194557 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194422    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-root\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194557 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194457    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194557 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194493    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-textfile\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194557 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194497    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-wtmp\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194557 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194514    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-sys\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194810 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194560    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-sys\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194810 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194563    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-tls\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194810 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194603    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17f1df83-bba7-4eae-8a25-bfb807a364c5-root\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.194964 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.194944    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-textfile\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.195096 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.195070    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.195185 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.195145    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f1df83-bba7-4eae-8a25-bfb807a364c5-metrics-client-ca\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.197060 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.197041    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.197145 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.197130    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17f1df83-bba7-4eae-8a25-bfb807a364c5-node-exporter-tls\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.246967 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.246945    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mc9\" (UniqueName: \"kubernetes.io/projected/17f1df83-bba7-4eae-8a25-bfb807a364c5-kube-api-access-l7mc9\") pod \"node-exporter-mpbk9\" (UID: \"17f1df83-bba7-4eae-8a25-bfb807a364c5\") " pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.425272 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.425196    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpbk9"
Nov 03 11:51:23.451664 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:23.451636    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f1df83_bba7_4eae_8a25_bfb807a364c5.slice/crio-995fb12ee6580000d74a9127fcf60362626a0de808e5b002ddfb5b31607c7e05 WatchSource:0}: Error finding container 995fb12ee6580000d74a9127fcf60362626a0de808e5b002ddfb5b31607c7e05: Status 404 returned error can't find the container with id 995fb12ee6580000d74a9127fcf60362626a0de808e5b002ddfb5b31607c7e05
Nov 03 11:51:23.721615 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:23.721578    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpbk9" event={"ID":"17f1df83-bba7-4eae-8a25-bfb807a364c5","Type":"ContainerStarted","Data":"995fb12ee6580000d74a9127fcf60362626a0de808e5b002ddfb5b31607c7e05"}
Nov 03 11:51:26.737836 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:26.737802    4546 generic.go:356] "Generic (PLEG): container finished" podID="17f1df83-bba7-4eae-8a25-bfb807a364c5" containerID="5201a18d3a1e3816bf7041d0de87cbfff9570f14ed603d49682611990a04398e" exitCode=0
Nov 03 11:51:26.738270 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:26.737866    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpbk9" event={"ID":"17f1df83-bba7-4eae-8a25-bfb807a364c5","Type":"ContainerDied","Data":"5201a18d3a1e3816bf7041d0de87cbfff9570f14ed603d49682611990a04398e"}
Nov 03 11:51:27.740571 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.740538    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-b424q"]
Nov 03 11:51:27.753590 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.753559    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpbk9" event={"ID":"17f1df83-bba7-4eae-8a25-bfb807a364c5","Type":"ContainerStarted","Data":"ec73f8e17c3d36c6f3acab8d6b820d945edb080a2c41048c249121041b46ed3c"}
Nov 03 11:51:27.753726 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.753594    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpbk9" event={"ID":"17f1df83-bba7-4eae-8a25-bfb807a364c5","Type":"ContainerStarted","Data":"f0a45afdd8b09d34efef1838073b3789c1b8c4f750bcba8b6c809fabd28cdd9f"}
Nov 03 11:51:27.753726 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.753672    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:27.833211 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.833184    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjf9c\" (UniqueName: \"kubernetes.io/projected/e81db497-fb6d-4f0f-9568-6eb450d61851-kube-api-access-vjf9c\") pod \"monitoring-plugin-77f4f76594-b424q\" (UID: \"e81db497-fb6d-4f0f-9568-6eb450d61851\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:27.833390 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.833243    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e81db497-fb6d-4f0f-9568-6eb450d61851-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b424q\" (UID: \"e81db497-fb6d-4f0f-9568-6eb450d61851\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:27.840062 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.840037    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fd865\""
Nov 03 11:51:27.840191 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.840119    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 11:51:27.840857 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.840838    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-b424q"]
Nov 03 11:51:27.900286 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.900233    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mpbk9" podStartSLOduration=2.224013496 podStartE2EDuration="4.900217508s" totalImagesPullingTime="2.579296708s" totalInitContainerRuntime="96.907304ms" isStatefulPod=true podCreationTimestamp="2026-11-03 11:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:27.900211716 +0000 UTC m=+29.471795629" watchObservedRunningTime="2026-11-03 11:51:27.900217508 +0000 UTC m=+29.471801417"
Nov 03 11:51:27.934640 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.934611    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjf9c\" (UniqueName: \"kubernetes.io/projected/e81db497-fb6d-4f0f-9568-6eb450d61851-kube-api-access-vjf9c\") pod \"monitoring-plugin-77f4f76594-b424q\" (UID: \"e81db497-fb6d-4f0f-9568-6eb450d61851\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:27.934810 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.934664    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e81db497-fb6d-4f0f-9568-6eb450d61851-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b424q\" (UID: \"e81db497-fb6d-4f0f-9568-6eb450d61851\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:27.937438 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.937413    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e81db497-fb6d-4f0f-9568-6eb450d61851-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b424q\" (UID: \"e81db497-fb6d-4f0f-9568-6eb450d61851\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:27.988424 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:27.988398    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjf9c\" (UniqueName: \"kubernetes.io/projected/e81db497-fb6d-4f0f-9568-6eb450d61851-kube-api-access-vjf9c\") pod \"monitoring-plugin-77f4f76594-b424q\" (UID: \"e81db497-fb6d-4f0f-9568-6eb450d61851\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:28.081451 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:28.081378    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:28.468770 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:28.468721    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-b424q"]
Nov 03 11:51:28.472553 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:28.472512    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81db497_fb6d_4f0f_9568_6eb450d61851.slice/crio-bbb000cc538534b65ef8ff33281450324cd56d1925d030cdbbacd79770d30809 WatchSource:0}: Error finding container bbb000cc538534b65ef8ff33281450324cd56d1925d030cdbbacd79770d30809: Status 404 returned error can't find the container with id bbb000cc538534b65ef8ff33281450324cd56d1925d030cdbbacd79770d30809
Nov 03 11:51:28.749670 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:28.749580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q" event={"ID":"e81db497-fb6d-4f0f-9568-6eb450d61851","Type":"ContainerStarted","Data":"bbb000cc538534b65ef8ff33281450324cd56d1925d030cdbbacd79770d30809"}
Nov 03 11:51:31.408886 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.408853    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pgfhl"
Nov 03 11:51:31.762431 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.762385    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q" event={"ID":"e81db497-fb6d-4f0f-9568-6eb450d61851","Type":"ContainerStarted","Data":"329a2effebc85a085361c14440d61006567aefcb61ab04e2cdcf1db44e81abf1"}
Nov 03 11:51:31.762627 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.762614    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:31.764529 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.764497    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sbqz8" event={"ID":"75caa47e-e5fd-4bf0-96fb-ddcf53ea3745","Type":"ContainerStarted","Data":"0db2a6b29506ef4fc8ada9ecb4c41cc4173e6c7754a5ebc40ae8f4953f6cd5f0"}
Nov 03 11:51:31.765186 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.765167    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:31.769153 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.769125    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q"
Nov 03 11:51:31.779798 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.779773    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-sbqz8"
Nov 03 11:51:31.861139 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.861104    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:51:31.864616 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.864591    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.913431 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.913402    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 11:51:31.913599 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.913496    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 11:51:31.913599 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.913508    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 11:51:31.916224 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.916202    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1u4oldrtc1tfg\""
Nov 03 11:51:31.917179 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.917135    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b424q" podStartSLOduration=2.564660909 podStartE2EDuration="4.91712181s" totalImagesPullingTime="2.352460901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:31.916633606 +0000 UTC m=+33.488217541" watchObservedRunningTime="2026-11-03 11:51:31.91712181 +0000 UTC m=+33.488705722"
Nov 03 11:51:31.960071 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.960038    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 11:51:31.960071 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.960056    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 11:51:31.960287 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.960102    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 11:51:31.960287 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.960210    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 11:51:31.961427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.961393    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:51:31.969870 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.969843    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.969991 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.969890    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.969991 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.969918    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970102 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970056    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970202 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970179    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970251 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970210    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970251 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970230    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970251 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970248    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970399 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970323    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970451 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970407    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970506 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970487    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970560 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970526    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970598 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970564    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970650 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970634    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970689 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970670    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970741 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970699    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970810 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970792    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970862 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970822    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970862 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970841    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:31.970952 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:31.970881    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmvl\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.052456 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.052365    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 11:51:32.052809 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.052772    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 11:51:32.055305 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055281    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 11:51:32.055427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055323    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 11:51:32.055427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055408    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 11:51:32.055533 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055451    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gjt9p\""
Nov 03 11:51:32.055594 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055534    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 11:51:32.055642 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055594    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 11:51:32.055702 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.055688    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 11:51:32.071507 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071480    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.071620 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071525    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.071683 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071648    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.071741 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071687    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.071855 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071837    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.071929 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071875    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.071929 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071899    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmvl\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072021 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071936    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072021 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071967    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072021 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.071994    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072172 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072024    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072172 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072054    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072172 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072077    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072172 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072109    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072172 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072139    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072406 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072195    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072406 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072224    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072406 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072244    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072406 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072257    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072406 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072316    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072406 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072355    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072693 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072574    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072693 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072674    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.072830 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.072803    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.082230 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.075469    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.082230 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.076271    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.082230 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.079328    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.082230 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.080733    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.086250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.083230    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.086250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.083592    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.090918 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.090485    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.091306 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.091287    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.093640 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.093613    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.095567 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.095543    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.097438 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.097379    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.097695 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.097669    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.097945 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.097925    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.099152 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.099117    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.099719 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.099693    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.129579 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.129547    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmvl\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl\") pod \"prometheus-k8s-0\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.185098 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.185067    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:51:32.330479 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.330369    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-sbqz8" podStartSLOduration=2.366320591 podStartE2EDuration="20.330349756s" totalImagesPullingTime="17.964029165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:32.329434732 +0000 UTC m=+33.901018670" watchObservedRunningTime="2026-11-03 11:51:32.330349756 +0000 UTC m=+33.901933670"
Nov 03 11:51:32.605433 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.605194    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:51:32.608421 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:51:32.608391    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e7bc21_ca77_4030_8e8b_2cbee91a5eb2.slice/crio-0d6eed43569cecfd0a569b5c3fe312f7270bb917dcb680d027ed986be1394271 WatchSource:0}: Error finding container 0d6eed43569cecfd0a569b5c3fe312f7270bb917dcb680d027ed986be1394271: Status 404 returned error can't find the container with id 0d6eed43569cecfd0a569b5c3fe312f7270bb917dcb680d027ed986be1394271
Nov 03 11:51:32.770080 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:32.770035    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"0d6eed43569cecfd0a569b5c3fe312f7270bb917dcb680d027ed986be1394271"}
Nov 03 11:51:35.700392 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:35.700361    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b5cc5f87-cxg2p"
Nov 03 11:51:35.784247 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:35.784209    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41" exitCode=0
Nov 03 11:51:35.784427 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:35.784270    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41"}
Nov 03 11:51:37.660661 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:51:37.660459    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:51:40.805912 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:40.805872    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1"}
Nov 03 11:51:40.806361 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:40.805941    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3"}
Nov 03 11:51:43.820115 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:43.820087    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350"}
Nov 03 11:51:44.826129 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:44.826094    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb"}
Nov 03 11:51:44.826609 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:44.826173    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb"}
Nov 03 11:51:44.826609 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:44.826215    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerStarted","Data":"4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413"}
Nov 03 11:51:45.079689 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:45.079577    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.273821363 podStartE2EDuration="14.079563332s" totalImagesPullingTime="9.790490345s" totalInitContainerRuntime="15.251624ms" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:45.033216174 +0000 UTC m=+46.604800086" watchObservedRunningTime="2026-11-03 11:51:45.079563332 +0000 UTC m=+46.651147244"
Nov 03 11:51:47.186170 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:51:47.186140    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:04.946540 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:04.946507    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:04.996140 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:04.996117    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:05.919613 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:05.919585    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:07.650450 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:07.650411    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:52:13.498312 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.498283    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gs7fz-8qhqm"]
Nov 03 11:52:13.498721 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.498667    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm" podUID="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" containerName="insights-gathering" containerID="cri-o://be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832" gracePeriod=30
Nov 03 11:52:13.785107 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.785079    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:52:13.795938 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.795917    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") pod \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") "
Nov 03 11:52:13.796045 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.795951    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") pod \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") "
Nov 03 11:52:13.796045 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.795978    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") pod \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") "
Nov 03 11:52:13.796045 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.796011    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") pod \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\" (UID: \"de76cb62-4bfe-40a0-8b30-2de34cb21dc9\") "
Nov 03 11:52:13.796256 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.796233    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path" pod "de76cb62-4bfe-40a0-8b30-2de34cb21dc9" (UID: "de76cb62-4bfe-40a0-8b30-2de34cb21dc9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:13.796328 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.796307    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle" pod "de76cb62-4bfe-40a0-8b30-2de34cb21dc9" (UID: "de76cb62-4bfe-40a0-8b30-2de34cb21dc9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:13.796415 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.796400    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle" pod "de76cb62-4bfe-40a0-8b30-2de34cb21dc9" (UID: "de76cb62-4bfe-40a0-8b30-2de34cb21dc9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:13.798379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.798361    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p" pod "de76cb62-4bfe-40a0-8b30-2de34cb21dc9" (UID: "de76cb62-4bfe-40a0-8b30-2de34cb21dc9"). InnerVolumeSpecName "kube-api-access-q268p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:13.897113 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.897080    4546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-trusted-ca-bundle\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:13.897113 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.897111    4546 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-service-ca-bundle\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:13.897304 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.897123    4546 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-archives-path\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:13.897304 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.897137    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q268p\" (UniqueName: \"kubernetes.io/projected/de76cb62-4bfe-40a0-8b30-2de34cb21dc9-kube-api-access-q268p\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:13.938491 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.938457    4546 generic.go:356] "Generic (PLEG): container finished" podID="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" containerID="be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832" exitCode=2
Nov 03 11:52:13.938664 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.938512    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm"
Nov 03 11:52:13.938664 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.938544    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm" event={"ID":"de76cb62-4bfe-40a0-8b30-2de34cb21dc9","Type":"ContainerDied","Data":"be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832"}
Nov 03 11:52:13.938664 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.938602    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gs7fz-8qhqm" event={"ID":"de76cb62-4bfe-40a0-8b30-2de34cb21dc9","Type":"ContainerDied","Data":"3e1672a39e89fee7b509946c07949a123194ef8ad1247d5e08f96676a746940d"}
Nov 03 11:52:13.938664 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.938626    4546 scope.go:122] "RemoveContainer" containerID="be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832"
Nov 03 11:52:13.947449 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.947432    4546 scope.go:122] "RemoveContainer" containerID="be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832"
Nov 03 11:52:13.947711 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:13.947682    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832\": container with ID starting with be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832 not found: ID does not exist" containerID="be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832"
Nov 03 11:52:13.947796 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:13.947727    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832"} err="failed to get container status \"be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832\": rpc error: code = NotFound desc = could not find container \"be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832\": container with ID starting with be047b0a677bbab3ae52cbd2a2760b46ac6db11ec10e979c1b87c52bd7a82832 not found: ID does not exist"
Nov 03 11:52:14.093910 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:14.092409    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gs7fz-8qhqm"]
Nov 03 11:52:14.095314 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:14.095282    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-gs7fz-8qhqm"]
Nov 03 11:52:14.570256 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:14.570227    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" path="/var/lib/kubelet/pods/de76cb62-4bfe-40a0-8b30-2de34cb21dc9/volumes"
Nov 03 11:52:23.887820 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.887786    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:52:23.888388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.888138    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="prometheus" containerID="cri-o://70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3" gracePeriod=600
Nov 03 11:52:23.888388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.888175    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy" containerID="cri-o://2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb" gracePeriod=600
Nov 03 11:52:23.888388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.888201    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="thanos-sidecar" containerID="cri-o://9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350" gracePeriod=600
Nov 03 11:52:23.888388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.888224    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="config-reloader" containerID="cri-o://2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1" gracePeriod=600
Nov 03 11:52:23.888388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.888285    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-web" containerID="cri-o://4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413" gracePeriod=600
Nov 03 11:52:23.888388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:23.888243    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb" gracePeriod=600
Nov 03 11:52:24.978652 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:24.978625    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413" exitCode=0
Nov 03 11:52:24.978652 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:24.978649    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350" exitCode=0
Nov 03 11:52:24.979018 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:24.978659    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1" exitCode=0
Nov 03 11:52:24.979018 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:24.978678    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413"}
Nov 03 11:52:24.979018 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:24.978778    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350"}
Nov 03 11:52:24.979018 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:24.978806    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1"}
Nov 03 11:52:25.904147 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:25.904102    4546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 11:52:25.904329 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:25.904171    4546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 11:52:25.904329 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:25.904193    4546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 11:52:25.904329 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:25.904212    4546 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 11:52:25.985714 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:25.985689    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb" exitCode=0
Nov 03 11:52:25.985714 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:25.985709    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb" exitCode=0
Nov 03 11:52:25.986075 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:25.985727    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb"}
Nov 03 11:52:25.986075 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:25.985796    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb"}
Nov 03 11:52:30.437244 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.437222    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:30.528135 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528105    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528135 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528139    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528162    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528182    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528202    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528236    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528283    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528320    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528358 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528345    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528378    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528406    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528439    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528470    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528502    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528532    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528578    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528624    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.528687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528667    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.529379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528695    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.529379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528723    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file\") pod \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\" (UID: \"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2\") "
Nov 03 11:52:30.529379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.528836    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.529379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.529146    4546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.529575 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.529480    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:30.531384 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.529795    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.531384 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.529852    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.531384 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.530205    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.531384 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.530552    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.531660 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.531522    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "kube-api-access-hhmvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:30.531820 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.531705    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.532157 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.532122    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:30.532354 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.532321    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:30.532500 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.532417    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.532714 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.532689    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.533120 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.533089    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:30.533832 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.533797    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.534275 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.534238    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.534275 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.534258    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.534422 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.534383    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.534826 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.534803    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.535526 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.535504    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.544051 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.544030    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config" pod "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" (UID: "25e7bc21-ca77-4030-8e8b-2cbee91a5eb2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:30.629714 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629684    4546 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629714 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629708    4546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629720    4546 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-metrics-client-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629733    4546 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629742    4546 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-kube-rbac-proxy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629778    4546 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629787    4546 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629801    4546 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-grpc-tls\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629813    4546 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629822    4546 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629830    4546 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629839    4546 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-config-out\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629847    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhmvl\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-kube-api-access-hhmvl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629855    4546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-db\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629862    4546 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-tls-assets\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.629874 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629870    4546 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.630286 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629899    4546 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-web-config\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.630286 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629908    4546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:30.630286 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:30.629916    4546 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2-secret-metrics-client-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:31.004511 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.004474    4546 generic.go:356] "Generic (PLEG): container finished" podID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3" exitCode=0
Nov 03 11:52:31.004696 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.004528    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3"}
Nov 03 11:52:31.004696 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.004552    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.004696 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.004586    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25e7bc21-ca77-4030-8e8b-2cbee91a5eb2","Type":"ContainerDied","Data":"0d6eed43569cecfd0a569b5c3fe312f7270bb917dcb680d027ed986be1394271"}
Nov 03 11:52:31.004696 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.004619    4546 scope.go:122] "RemoveContainer" containerID="e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb"
Nov 03 11:52:31.012138 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.012122    4546 scope.go:122] "RemoveContainer" containerID="2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb"
Nov 03 11:52:31.018644 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.018630    4546 scope.go:122] "RemoveContainer" containerID="4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413"
Nov 03 11:52:31.024599 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.024583    4546 scope.go:122] "RemoveContainer" containerID="9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350"
Nov 03 11:52:31.030525 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.030505    4546 scope.go:122] "RemoveContainer" containerID="2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1"
Nov 03 11:52:31.036569 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.036553    4546 scope.go:122] "RemoveContainer" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3"
Nov 03 11:52:31.043053 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.043037    4546 scope.go:122] "RemoveContainer" containerID="52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41"
Nov 03 11:52:31.049051 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049035    4546 scope.go:122] "RemoveContainer" containerID="e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb"
Nov 03 11:52:31.049341 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.049318    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb\": container with ID starting with e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb not found: ID does not exist" containerID="e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb"
Nov 03 11:52:31.049404 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049353    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb"} err="failed to get container status \"e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb\": rpc error: code = NotFound desc = could not find container \"e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb\": container with ID starting with e354a26842d2e6f204f0917c974a59a8812549993de059706b591b0a0cb606eb not found: ID does not exist"
Nov 03 11:52:31.049404 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049371    4546 scope.go:122] "RemoveContainer" containerID="2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb"
Nov 03 11:52:31.049634 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.049613    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb\": container with ID starting with 2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb not found: ID does not exist" containerID="2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb"
Nov 03 11:52:31.049690 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049646    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb"} err="failed to get container status \"2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb\": rpc error: code = NotFound desc = could not find container \"2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb\": container with ID starting with 2f2ce84fbbcdaa7d71ee121a5fb5c78f698fd3a0a853ec9581291045875547eb not found: ID does not exist"
Nov 03 11:52:31.049690 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049666    4546 scope.go:122] "RemoveContainer" containerID="4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413"
Nov 03 11:52:31.049934 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.049917    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413\": container with ID starting with 4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413 not found: ID does not exist" containerID="4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413"
Nov 03 11:52:31.049993 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049940    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413"} err="failed to get container status \"4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413\": rpc error: code = NotFound desc = could not find container \"4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413\": container with ID starting with 4cc33c1be0dffef2710560bd88ea92b8cd18accb789a6d49683d76ccbf499413 not found: ID does not exist"
Nov 03 11:52:31.049993 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.049957    4546 scope.go:122] "RemoveContainer" containerID="9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350"
Nov 03 11:52:31.050247 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.050230    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350\": container with ID starting with 9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350 not found: ID does not exist" containerID="9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350"
Nov 03 11:52:31.050299 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050251    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350"} err="failed to get container status \"9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350\": rpc error: code = NotFound desc = could not find container \"9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350\": container with ID starting with 9071b7fd69a58106d6b9ef3d8905765541759ad82ff615ebafa14c7ec23e9350 not found: ID does not exist"
Nov 03 11:52:31.050299 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050264    4546 scope.go:122] "RemoveContainer" containerID="2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1"
Nov 03 11:52:31.050491 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.050476    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1\": container with ID starting with 2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1 not found: ID does not exist" containerID="2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1"
Nov 03 11:52:31.050561 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050496    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1"} err="failed to get container status \"2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1\": rpc error: code = NotFound desc = could not find container \"2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1\": container with ID starting with 2cdef4ac6b67d823aa566d7d01092f31079593f09ed558e814869edffd173ce1 not found: ID does not exist"
Nov 03 11:52:31.050561 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050509    4546 scope.go:122] "RemoveContainer" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3"
Nov 03 11:52:31.050720 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.050707    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3\": container with ID starting with 70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3 not found: ID does not exist" containerID="70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3"
Nov 03 11:52:31.050780 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050725    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3"} err="failed to get container status \"70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3\": rpc error: code = NotFound desc = could not find container \"70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3\": container with ID starting with 70450d0a4b73239a38f3eba5d0c88588c7f427cf0e00c11f10fcbc41568a1ca3 not found: ID does not exist"
Nov 03 11:52:31.050780 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050737    4546 scope.go:122] "RemoveContainer" containerID="52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41"
Nov 03 11:52:31.050968 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:31.050955    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41\": container with ID starting with 52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41 not found: ID does not exist" containerID="52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41"
Nov 03 11:52:31.051002 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.050972    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41"} err="failed to get container status \"52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41\": rpc error: code = NotFound desc = could not find container \"52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41\": container with ID starting with 52ca31644a6064ceb04fa73d1c94aade8a11b6b8e4f114bcd30b004a36d2ab41 not found: ID does not exist"
Nov 03 11:52:31.161230 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.161204    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:52:31.167384 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.167369    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:52:31.207076 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207055    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:52:31.207278 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207260    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy"
Nov 03 11:52:31.207278 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207272    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207285    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="init-config-reloader"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207290    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="init-config-reloader"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207299    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="prometheus"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207304    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="prometheus"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207310    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-thanos"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207315    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-thanos"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207322    4546 cpu_manager.go:414] "containerMap: removing container" podUID="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" containerName="insights-gathering"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207326    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" containerName="insights-gathering"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207334    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="config-reloader"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207339    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="config-reloader"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207344    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="thanos-sidecar"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207348    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="thanos-sidecar"
Nov 03 11:52:31.207349 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207353    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-web"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207358    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-web"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207404    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="config-reloader"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207410    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="prometheus"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207416    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-thanos"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207422    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="de76cb62-4bfe-40a0-8b30-2de34cb21dc9" containerName="insights-gathering"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207427    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="thanos-sidecar"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207431    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy-web"
Nov 03 11:52:31.207698 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.207436    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" containerName="kube-rbac-proxy"
Nov 03 11:52:31.210009 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.209992    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.260250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260185    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 11:52:31.260250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260187    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 11:52:31.260250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260226    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 11:52:31.260250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260241    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 11:52:31.260250 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260241    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260275    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260305    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260336    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1u4oldrtc1tfg\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260470    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260491    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260513    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 11:52:31.260583 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260549    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gjt9p\""
Nov 03 11:52:31.260939 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260824    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 11:52:31.260939 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260849    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 11:52:31.260939 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.260884    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 11:52:31.307266 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.307242    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 11:52:31.307379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.307287    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 11:52:31.321950 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.321925    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:52:31.335290 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335264    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0161452-0240-42ac-8238-f85ec8057284-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335411 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335308    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335480 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335437    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335535 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335480    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335535 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335514    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-config\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335611 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335541    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335666 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335648    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ptg\" (UniqueName: \"kubernetes.io/projected/e0161452-0240-42ac-8238-f85ec8057284-kube-api-access-44ptg\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335712 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335682    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335772 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335739    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335837 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335822    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335890 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335870    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335937 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335903    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.335982 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.335966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336015 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336006    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336045 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336032    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336094 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336066    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336129 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336104    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336171 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336158    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336215 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336199    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0161452-0240-42ac-8238-f85ec8057284-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.336249 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.336227    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436695 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436662    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436695 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436696    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436718    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-config\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436742    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436791    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ptg\" (UniqueName: \"kubernetes.io/projected/e0161452-0240-42ac-8238-f85ec8057284-kube-api-access-44ptg\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436812    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436847    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436873    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436893    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.436921 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436919    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436954    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.436987    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437015    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437041    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437066    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437093    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437127    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0161452-0240-42ac-8238-f85ec8057284-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437155    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437185    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0161452-0240-42ac-8238-f85ec8057284-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.437285 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437210    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.438540 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437295    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.438540 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437389    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.438540 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437649    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.438540 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.437835    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.438540 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.438157    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.439004 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.438978    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.439260 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.439232    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.440362 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.440290    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.441511 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.441440    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-config\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.441511 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.441455    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0161452-0240-42ac-8238-f85ec8057284-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.441511 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.441469    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.441885 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.441862    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.441975 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.441932    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.442431 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.442396    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0161452-0240-42ac-8238-f85ec8057284-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.442521 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.442471    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.442521 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.442477    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.442865 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.442847    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0161452-0240-42ac-8238-f85ec8057284-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.442991 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.442974    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.443619 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.443602    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0161452-0240-42ac-8238-f85ec8057284-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.490334 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.490311    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ptg\" (UniqueName: \"kubernetes.io/projected/e0161452-0240-42ac-8238-f85ec8057284-kube-api-access-44ptg\") pod \"prometheus-k8s-0\" (UID: \"e0161452-0240-42ac-8238-f85ec8057284\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.531502 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.531439    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:31.943134 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:31.942983    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 11:52:31.945214 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:52:31.945189    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0161452_0240_42ac_8238_f85ec8057284.slice/crio-510e7889808ba7ccb407b0918ab8aa43159e8797ddfd4ee655de7afc36b61b22 WatchSource:0}: Error finding container 510e7889808ba7ccb407b0918ab8aa43159e8797ddfd4ee655de7afc36b61b22: Status 404 returned error can't find the container with id 510e7889808ba7ccb407b0918ab8aa43159e8797ddfd4ee655de7afc36b61b22
Nov 03 11:52:32.011066 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:32.011035    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"8ccb84fd583bda3fbd51bc1f554952db23f2af675c36eb715a3bb9b32c665b4b"}
Nov 03 11:52:32.011237 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:32.011109    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"510e7889808ba7ccb407b0918ab8aa43159e8797ddfd4ee655de7afc36b61b22"}
Nov 03 11:52:32.571142 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:32.571101    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e7bc21-ca77-4030-8e8b-2cbee91a5eb2" path="/var/lib/kubelet/pods/25e7bc21-ca77-4030-8e8b-2cbee91a5eb2/volumes"
Nov 03 11:52:33.016224 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:33.016193    4546 generic.go:356] "Generic (PLEG): container finished" podID="e0161452-0240-42ac-8238-f85ec8057284" containerID="8ccb84fd583bda3fbd51bc1f554952db23f2af675c36eb715a3bb9b32c665b4b" exitCode=0
Nov 03 11:52:33.016388 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:33.016230    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerDied","Data":"8ccb84fd583bda3fbd51bc1f554952db23f2af675c36eb715a3bb9b32c665b4b"}
Nov 03 11:52:34.025870 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.025841    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"da36d427be4289ca55732f8ebda17953518675f342e48a92341541fa07c39217"}
Nov 03 11:52:34.026239 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.025920    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"58d506fd5ba7037ce06ca36b40c1f837ad6b6c4c7275ea83b280dbb0483ac4ae"}
Nov 03 11:52:34.026239 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.025946    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"024fc43c1dfbb0e524042f0639b707cc350b808252b46e2814e5441d0f6ec166"}
Nov 03 11:52:34.026239 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.025973    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"45091c16ae85529915dc2973cd2fea97db103c458852b612923a57156d3c96ec"}
Nov 03 11:52:34.026239 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.026011    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"b2b29c38eb672e4f92aa2cd9d5447ca6da9a188d834de0d3f0d1852be7fb701f"}
Nov 03 11:52:34.026239 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.026037    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0161452-0240-42ac-8238-f85ec8057284","Type":"ContainerStarted","Data":"311580fb96f536b86f0321777b7f568fe613d7df6fded8ad5a36da2ff363ad42"}
Nov 03 11:52:34.149884 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:34.149839    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.13053163 podStartE2EDuration="3.149826335s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.294705ms" isStatefulPod=false podCreationTimestamp="2026-11-03 11:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:52:34.148234169 +0000 UTC m=+95.719818093" watchObservedRunningTime="2026-11-03 11:52:34.149826335 +0000 UTC m=+95.721410245"
Nov 03 11:52:36.531856 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:52:36.531826    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:52:37.648523 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:52:37.648489    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:53:07.648563 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:53:07.648477    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:53:31.532356 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:53:31.532321    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:53:31.547833 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:53:31.547809    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:53:32.231990 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:53:32.231966    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 11:53:37.649069 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:53:37.649032    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:54:07.649336 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:54:07.649301    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:54:37.648992 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:54:37.648906    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:55:07.647444 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:55:07.647414    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:55:37.650054 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:55:37.650009    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:56:07.649668 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:56:07.649591    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:56:37.649367 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:56:37.649332    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:56:44.692545 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.692496    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"]
Nov 03 11:56:44.697786 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.697739    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.802124 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.802090    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gt24\" (UniqueName: \"kubernetes.io/projected/90607d76-18a7-4d47-8b68-404cd07bc3c0-kube-api-access-6gt24\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.802309 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.802132    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90607d76-18a7-4d47-8b68-404cd07bc3c0-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.802309 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.802162    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.802309 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.802289    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90607d76-18a7-4d47-8b68-404cd07bc3c0-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.802440 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.802321    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.802440 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.802342    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.821798 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.821776    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 11:56:44.821924 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.821840    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 11:56:44.821924 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.821892    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 11:56:44.821924 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.821886    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 11:56:44.822051 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.821962    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zcjpl\""
Nov 03 11:56:44.822051 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.822028    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 11:56:44.903544 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903506    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90607d76-18a7-4d47-8b68-404cd07bc3c0-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.903544 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903546    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.903836 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903563    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.903836 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903618    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gt24\" (UniqueName: \"kubernetes.io/projected/90607d76-18a7-4d47-8b68-404cd07bc3c0-kube-api-access-6gt24\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.903836 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903633    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90607d76-18a7-4d47-8b68-404cd07bc3c0-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.903836 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903652    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.904026 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.903992    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90607d76-18a7-4d47-8b68-404cd07bc3c0-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.906213 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.906187    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90607d76-18a7-4d47-8b68-404cd07bc3c0-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.906318 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.906251    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.906367 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.906351    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.906543 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.906524    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90607d76-18a7-4d47-8b68-404cd07bc3c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:44.928574 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.928549    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"]
Nov 03 11:56:44.960357 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:44.960298    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gt24\" (UniqueName: \"kubernetes.io/projected/90607d76-18a7-4d47-8b68-404cd07bc3c0-kube-api-access-6gt24\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-w5hwz\" (UID: \"90607d76-18a7-4d47-8b68-404cd07bc3c0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:45.013117 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:45.013089    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:45.478382 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:45.478248    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"]
Nov 03 11:56:45.484054 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:56:45.484027    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90607d76_18a7_4d47_8b68_404cd07bc3c0.slice/crio-af090ccd98afe2a9e2608bf75d7009ab37a9a1992aded21e1578c973db6e77e9 WatchSource:0}: Error finding container af090ccd98afe2a9e2608bf75d7009ab37a9a1992aded21e1578c973db6e77e9: Status 404 returned error can't find the container with id af090ccd98afe2a9e2608bf75d7009ab37a9a1992aded21e1578c973db6e77e9
Nov 03 11:56:45.485633 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:45.485617    4546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:56:45.837719 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:45.837642    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz" event={"ID":"90607d76-18a7-4d47-8b68-404cd07bc3c0","Type":"ContainerStarted","Data":"af090ccd98afe2a9e2608bf75d7009ab37a9a1992aded21e1578c973db6e77e9"}
Nov 03 11:56:52.864445 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:52.864412    4546 generic.go:356] "Generic (PLEG): container finished" podID="90607d76-18a7-4d47-8b68-404cd07bc3c0" containerID="c086023706a77f8e04d003044d299feb6217e37af64a1a902a91b167501a6227" exitCode=0
Nov 03 11:56:52.864975 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:52.864465    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz" event={"ID":"90607d76-18a7-4d47-8b68-404cd07bc3c0","Type":"ContainerDied","Data":"c086023706a77f8e04d003044d299feb6217e37af64a1a902a91b167501a6227"}
Nov 03 11:56:53.869849 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:53.869816    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz" event={"ID":"90607d76-18a7-4d47-8b68-404cd07bc3c0","Type":"ContainerStarted","Data":"5a4328fa7934e6a5751ec6380a507d471b29a27f0eb466f45b7b0fdf8996969f"}
Nov 03 11:56:53.870232 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:53.869978    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:56:54.119781 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:56:54.119714    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz" podStartSLOduration=2.558635995 podStartE2EDuration="10.119700942s" totalImagesPullingTime="7.150504636s" totalInitContainerRuntime="410.560311ms" isStatefulPod=false podCreationTimestamp="2026-11-03 11:56:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:54.073710545 +0000 UTC m=+355.645294476" watchObservedRunningTime="2026-11-03 11:56:54.119700942 +0000 UTC m=+355.691284851"
Nov 03 11:57:04.148558 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.148524    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"]
Nov 03 11:57:04.158630 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.158597    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.210928 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.210892    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"]
Nov 03 11:57:04.250964 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.250934    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a118d42-600f-4c2f-98eb-c6d9ec38776a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t5gw2\" (UID: \"5a118d42-600f-4c2f-98eb-c6d9ec38776a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.251088 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.250986    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w646p\" (UniqueName: \"kubernetes.io/projected/5a118d42-600f-4c2f-98eb-c6d9ec38776a-kube-api-access-w646p\") pod \"cert-manager-webhook-5dbbbf75cc-t5gw2\" (UID: \"5a118d42-600f-4c2f-98eb-c6d9ec38776a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.256025 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.256007    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 11:57:04.256077 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.256029    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 11:57:04.256107 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.256085    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j9plz\""
Nov 03 11:57:04.351663 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.351641    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a118d42-600f-4c2f-98eb-c6d9ec38776a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t5gw2\" (UID: \"5a118d42-600f-4c2f-98eb-c6d9ec38776a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.351818 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.351676    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w646p\" (UniqueName: \"kubernetes.io/projected/5a118d42-600f-4c2f-98eb-c6d9ec38776a-kube-api-access-w646p\") pod \"cert-manager-webhook-5dbbbf75cc-t5gw2\" (UID: \"5a118d42-600f-4c2f-98eb-c6d9ec38776a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.409744 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.409687    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w646p\" (UniqueName: \"kubernetes.io/projected/5a118d42-600f-4c2f-98eb-c6d9ec38776a-kube-api-access-w646p\") pod \"cert-manager-webhook-5dbbbf75cc-t5gw2\" (UID: \"5a118d42-600f-4c2f-98eb-c6d9ec38776a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.409865 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.409833    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a118d42-600f-4c2f-98eb-c6d9ec38776a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t5gw2\" (UID: \"5a118d42-600f-4c2f-98eb-c6d9ec38776a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.484379 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.484358    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:04.846514 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.846410    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"]
Nov 03 11:57:04.849212 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:04.849178    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a118d42_600f_4c2f_98eb_c6d9ec38776a.slice/crio-76fee73da0c8794160b14113d75f667d84c16221417f8b1caf8825dda070a09f WatchSource:0}: Error finding container 76fee73da0c8794160b14113d75f667d84c16221417f8b1caf8825dda070a09f: Status 404 returned error can't find the container with id 76fee73da0c8794160b14113d75f667d84c16221417f8b1caf8825dda070a09f
Nov 03 11:57:04.874681 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.874652    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-w5hwz"
Nov 03 11:57:04.904894 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:04.904865    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2" event={"ID":"5a118d42-600f-4c2f-98eb-c6d9ec38776a","Type":"ContainerStarted","Data":"76fee73da0c8794160b14113d75f667d84c16221417f8b1caf8825dda070a09f"}
Nov 03 11:57:07.657321 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:57:07.657283    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:57:08.317539 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.316111    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"]
Nov 03 11:57:08.323063 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.323035    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.373646 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.373611    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 11:57:08.383162 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.383126    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.383330 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.383183    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.383393 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.383367    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdm8\" (UniqueName: \"kubernetes.io/projected/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-kube-api-access-ztdm8\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.419764 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.419720    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 11:57:08.419942 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.419883    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 11:57:08.419942 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.419885    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 11:57:08.419942 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.419884    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8xttl\""
Nov 03 11:57:08.420884 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.420865    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"]
Nov 03 11:57:08.484650 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.484614    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdm8\" (UniqueName: \"kubernetes.io/projected/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-kube-api-access-ztdm8\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.484863 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.484673    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.484863 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.484704    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.485472 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.485449    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.487623 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.487599    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.541987 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.541954    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdm8\" (UniqueName: \"kubernetes.io/projected/b5f43aa3-c9b8-4f13-bfa7-d875cb364992-kube-api-access-ztdm8\") pod \"kuadrant-console-plugin-66c4fb575d-7749m\" (UID: \"b5f43aa3-c9b8-4f13-bfa7-d875cb364992\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:08.638503 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:08.638419    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"
Nov 03 11:57:09.011994 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:09.011925    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m"]
Nov 03 11:57:09.015257 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:09.015220    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5f43aa3_c9b8_4f13_bfa7_d875cb364992.slice/crio-2094dac50918da75a9bc30ba85eebfead30bfe1d4a3aab5d8a1a0aa0add47f65 WatchSource:0}: Error finding container 2094dac50918da75a9bc30ba85eebfead30bfe1d4a3aab5d8a1a0aa0add47f65: Status 404 returned error can't find the container with id 2094dac50918da75a9bc30ba85eebfead30bfe1d4a3aab5d8a1a0aa0add47f65
Nov 03 11:57:09.924900 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:09.924854    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m" event={"ID":"b5f43aa3-c9b8-4f13-bfa7-d875cb364992","Type":"ContainerStarted","Data":"2094dac50918da75a9bc30ba85eebfead30bfe1d4a3aab5d8a1a0aa0add47f65"}
Nov 03 11:57:09.926281 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:09.926262    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2" event={"ID":"5a118d42-600f-4c2f-98eb-c6d9ec38776a","Type":"ContainerStarted","Data":"ed4edb902e25188685a0183d207e62a88feecc0d81eac28253a719ee6d029a03"}
Nov 03 11:57:09.926416 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:09.926394    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:10.033244 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.033170    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2" podStartSLOduration=1.487807844 podStartE2EDuration="6.033152437s" totalImagesPullingTime="4.545344593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:10.033111732 +0000 UTC m=+371.604695655" watchObservedRunningTime="2026-11-03 11:57:10.033152437 +0000 UTC m=+371.604736348"
Nov 03 11:57:10.280555 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.280451    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"]
Nov 03 11:57:10.283843 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.283826    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.299380 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.299356    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e880b8-610e-493e-86fa-cd7c56e90e00-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dtl5x\" (UID: \"c4e880b8-610e-493e-86fa-cd7c56e90e00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.299500 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.299435    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srft7\" (UniqueName: \"kubernetes.io/projected/c4e880b8-610e-493e-86fa-cd7c56e90e00-kube-api-access-srft7\") pod \"cert-manager-cainjector-5c57bbd4fc-dtl5x\" (UID: \"c4e880b8-610e-493e-86fa-cd7c56e90e00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.332213 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.332190    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-knq2k\""
Nov 03 11:57:10.338552 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.338525    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"]
Nov 03 11:57:10.400191 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.400161    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srft7\" (UniqueName: \"kubernetes.io/projected/c4e880b8-610e-493e-86fa-cd7c56e90e00-kube-api-access-srft7\") pod \"cert-manager-cainjector-5c57bbd4fc-dtl5x\" (UID: \"c4e880b8-610e-493e-86fa-cd7c56e90e00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.400357 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.400206    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e880b8-610e-493e-86fa-cd7c56e90e00-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dtl5x\" (UID: \"c4e880b8-610e-493e-86fa-cd7c56e90e00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.454235 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.454207    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e880b8-610e-493e-86fa-cd7c56e90e00-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dtl5x\" (UID: \"c4e880b8-610e-493e-86fa-cd7c56e90e00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.454373 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.454263    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srft7\" (UniqueName: \"kubernetes.io/projected/c4e880b8-610e-493e-86fa-cd7c56e90e00-kube-api-access-srft7\") pod \"cert-manager-cainjector-5c57bbd4fc-dtl5x\" (UID: \"c4e880b8-610e-493e-86fa-cd7c56e90e00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.595446 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.595371    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"
Nov 03 11:57:10.957944 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:10.957711    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x"]
Nov 03 11:57:10.961149 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:10.961110    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4e880b8_610e_493e_86fa_cd7c56e90e00.slice/crio-657a0a7969529b6b14253f14939b6a63c2907299b250978dbb83b85369252cd5 WatchSource:0}: Error finding container 657a0a7969529b6b14253f14939b6a63c2907299b250978dbb83b85369252cd5: Status 404 returned error can't find the container with id 657a0a7969529b6b14253f14939b6a63c2907299b250978dbb83b85369252cd5
Nov 03 11:57:11.938222 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:11.937690    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x" event={"ID":"c4e880b8-610e-493e-86fa-cd7c56e90e00","Type":"ContainerStarted","Data":"8c07a4d7a8586e3c4be591c84a90a7386bb0ce5468ce39d696b2513f73ce0e8b"}
Nov 03 11:57:11.938222 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:11.937767    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x" event={"ID":"c4e880b8-610e-493e-86fa-cd7c56e90e00","Type":"ContainerStarted","Data":"657a0a7969529b6b14253f14939b6a63c2907299b250978dbb83b85369252cd5"}
Nov 03 11:57:12.044297 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:12.044234    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dtl5x" podStartSLOduration=2.044217689 podStartE2EDuration="2.044217689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:12.043985206 +0000 UTC m=+373.615569119" watchObservedRunningTime="2026-11-03 11:57:12.044217689 +0000 UTC m=+373.615801602"
Nov 03 11:57:14.951006 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:14.950914    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m" event={"ID":"b5f43aa3-c9b8-4f13-bfa7-d875cb364992","Type":"ContainerStarted","Data":"9784ef982b3438d6c81efac73a74088b57d54985931d3346a9d8e25b5aafddf3"}
Nov 03 11:57:15.056850 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:15.056647    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-7749m" podStartSLOduration=1.47278052 podStartE2EDuration="7.056630405s" totalImagesPullingTime="5.583849885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:15.055574959 +0000 UTC m=+376.627158870" watchObservedRunningTime="2026-11-03 11:57:15.056630405 +0000 UTC m=+376.628214320"
Nov 03 11:57:15.932854 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:15.932811    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2"
Nov 03 11:57:21.365341 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.363307    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"]
Nov 03 11:57:21.369550 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.369526    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.419842 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.419814    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 11:57:21.466276 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.466246    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 11:57:21.466276 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.466271    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 11:57:21.466522 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.466276    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 11:57:21.470452 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.470428    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"]
Nov 03 11:57:21.492660 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.492636    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6rw\" (UniqueName: \"kubernetes.io/projected/046066bc-e7e0-4fa0-8f15-f806f818a2d2-kube-api-access-lz6rw\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.492800 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.492684    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046066bc-e7e0-4fa0-8f15-f806f818a2d2-tls-certs\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.492800 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.492794    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046066bc-e7e0-4fa0-8f15-f806f818a2d2-tmp\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.520183 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.520161    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t7b7z\""
Nov 03 11:57:21.593348 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.593322    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046066bc-e7e0-4fa0-8f15-f806f818a2d2-tls-certs\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.593474 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.593361    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046066bc-e7e0-4fa0-8f15-f806f818a2d2-tmp\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.593474 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.593395    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6rw\" (UniqueName: \"kubernetes.io/projected/046066bc-e7e0-4fa0-8f15-f806f818a2d2-kube-api-access-lz6rw\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.595660 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.595637    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046066bc-e7e0-4fa0-8f15-f806f818a2d2-tmp\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.595805 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.595784    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046066bc-e7e0-4fa0-8f15-f806f818a2d2-tls-certs\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.649258 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.649191    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6rw\" (UniqueName: \"kubernetes.io/projected/046066bc-e7e0-4fa0-8f15-f806f818a2d2-kube-api-access-lz6rw\") pod \"kube-auth-proxy-758d88fbcf-s7zpk\" (UID: \"046066bc-e7e0-4fa0-8f15-f806f818a2d2\") " pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:21.681083 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:21.681057    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"
Nov 03 11:57:22.052044 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:22.052018    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk"]
Nov 03 11:57:22.054447 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:22.054421    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046066bc_e7e0_4fa0_8f15_f806f818a2d2.slice/crio-490999f3a81648034be6b8ad302588b7adc8a3ea3b602265c77d404ebdcdd60d WatchSource:0}: Error finding container 490999f3a81648034be6b8ad302588b7adc8a3ea3b602265c77d404ebdcdd60d: Status 404 returned error can't find the container with id 490999f3a81648034be6b8ad302588b7adc8a3ea3b602265c77d404ebdcdd60d
Nov 03 11:57:22.983065 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:22.983028    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk" event={"ID":"046066bc-e7e0-4fa0-8f15-f806f818a2d2","Type":"ContainerStarted","Data":"490999f3a81648034be6b8ad302588b7adc8a3ea3b602265c77d404ebdcdd60d"}
Nov 03 11:57:24.292460 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.292424    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-th82g"]
Nov 03 11:57:24.295865 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.295844    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.350305 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.350279    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-th82g"]
Nov 03 11:57:24.390585 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.390557    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7l542\""
Nov 03 11:57:24.417869 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.417828    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjlc\" (UniqueName: \"kubernetes.io/projected/81ce2ad3-30f4-4de6-bd7b-b551ba743476-kube-api-access-5xjlc\") pod \"cert-manager-597b95cb84-th82g\" (UID: \"81ce2ad3-30f4-4de6-bd7b-b551ba743476\") " pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.418054 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.417892    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81ce2ad3-30f4-4de6-bd7b-b551ba743476-bound-sa-token\") pod \"cert-manager-597b95cb84-th82g\" (UID: \"81ce2ad3-30f4-4de6-bd7b-b551ba743476\") " pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.518526 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.518495    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjlc\" (UniqueName: \"kubernetes.io/projected/81ce2ad3-30f4-4de6-bd7b-b551ba743476-kube-api-access-5xjlc\") pod \"cert-manager-597b95cb84-th82g\" (UID: \"81ce2ad3-30f4-4de6-bd7b-b551ba743476\") " pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.518706 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.518539    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81ce2ad3-30f4-4de6-bd7b-b551ba743476-bound-sa-token\") pod \"cert-manager-597b95cb84-th82g\" (UID: \"81ce2ad3-30f4-4de6-bd7b-b551ba743476\") " pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.573390 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.573295    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81ce2ad3-30f4-4de6-bd7b-b551ba743476-bound-sa-token\") pod \"cert-manager-597b95cb84-th82g\" (UID: \"81ce2ad3-30f4-4de6-bd7b-b551ba743476\") " pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.580530 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.580503    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjlc\" (UniqueName: \"kubernetes.io/projected/81ce2ad3-30f4-4de6-bd7b-b551ba743476-kube-api-access-5xjlc\") pod \"cert-manager-597b95cb84-th82g\" (UID: \"81ce2ad3-30f4-4de6-bd7b-b551ba743476\") " pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.610571 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.610366    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-th82g"
Nov 03 11:57:24.980655 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.980616    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-th82g"]
Nov 03 11:57:24.983867 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:24.983825    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81ce2ad3_30f4_4de6_bd7b_b551ba743476.slice/crio-299dcf06cc47b068bd81a312f2d799ff05a95ff7dbe42872ded32921a5ffd825 WatchSource:0}: Error finding container 299dcf06cc47b068bd81a312f2d799ff05a95ff7dbe42872ded32921a5ffd825: Status 404 returned error can't find the container with id 299dcf06cc47b068bd81a312f2d799ff05a95ff7dbe42872ded32921a5ffd825
Nov 03 11:57:24.992576 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:24.992543    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-th82g" event={"ID":"81ce2ad3-30f4-4de6-bd7b-b551ba743476","Type":"ContainerStarted","Data":"299dcf06cc47b068bd81a312f2d799ff05a95ff7dbe42872ded32921a5ffd825"}
Nov 03 11:57:25.997463 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:25.997429    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-th82g" event={"ID":"81ce2ad3-30f4-4de6-bd7b-b551ba743476","Type":"ContainerStarted","Data":"6e1cc10d9c3f5cc7b2c14cea03ff6ef6c840ea17e219d237ef27a6609fc46f18"}
Nov 03 11:57:26.106987 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.106918    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-th82g" podStartSLOduration=2.106900949 podStartE2EDuration="2.106900949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:26.105880241 +0000 UTC m=+387.677464154" watchObservedRunningTime="2026-11-03 11:57:26.106900949 +0000 UTC m=+387.678484857"
Nov 03 11:57:26.515857 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.515813    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:57:26.519297 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.519270    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.567769 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.567715    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 11:57:26.574101 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.574075    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:57:26.636135 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.636093    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9pc\" (UniqueName: \"kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc\") pod \"limitador-limitador-74dfcfb894-nrqtl\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.636135 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.636136    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file\") pod \"limitador-limitador-74dfcfb894-nrqtl\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.737116 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.737077    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9pc\" (UniqueName: \"kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc\") pod \"limitador-limitador-74dfcfb894-nrqtl\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.737116 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.737115    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file\") pod \"limitador-limitador-74dfcfb894-nrqtl\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.737687 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.737670    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file\") pod \"limitador-limitador-74dfcfb894-nrqtl\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.742721 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.742683    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:57:26.791019 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.790944    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9pc\" (UniqueName: \"kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc\") pod \"limitador-limitador-74dfcfb894-nrqtl\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:26.833090 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:26.833055    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:27.199479 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:27.199455    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:57:27.201684 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:27.201648    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789a7eb3_47cb_4717_9739_2516383aceb5.slice/crio-a3b9d9068c03a741421257806ace55ee06f86c67626d5d4f8436b7b12b9cb98c WatchSource:0}: Error finding container a3b9d9068c03a741421257806ace55ee06f86c67626d5d4f8436b7b12b9cb98c: Status 404 returned error can't find the container with id a3b9d9068c03a741421257806ace55ee06f86c67626d5d4f8436b7b12b9cb98c
Nov 03 11:57:28.006241 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:28.006205    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk" event={"ID":"046066bc-e7e0-4fa0-8f15-f806f818a2d2","Type":"ContainerStarted","Data":"9285e698b125f5258a2b78c4afcc4d639a382c8f3a166edcaaba23b4b4a22dfe"}
Nov 03 11:57:28.007401 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:28.007376    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl" event={"ID":"789a7eb3-47cb-4717-9739-2516383aceb5","Type":"ContainerStarted","Data":"a3b9d9068c03a741421257806ace55ee06f86c67626d5d4f8436b7b12b9cb98c"}
Nov 03 11:57:28.113708 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:28.113645    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-758d88fbcf-s7zpk" podStartSLOduration=1.9163449240000001 podStartE2EDuration="7.11362598s" totalImagesPullingTime="5.197281056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:28.112946674 +0000 UTC m=+389.684530623" watchObservedRunningTime="2026-11-03 11:57:28.11362598 +0000 UTC m=+389.685209895"
Nov 03 11:57:35.036476 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:35.036437    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl" event={"ID":"789a7eb3-47cb-4717-9739-2516383aceb5","Type":"ContainerStarted","Data":"041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e"}
Nov 03 11:57:35.037005 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:35.036513    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:35.142910 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:35.142847    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl" podStartSLOduration=1.737384211 podStartE2EDuration="9.142833802s" totalImagesPullingTime="7.405449591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:35.141892168 +0000 UTC m=+396.713476080" watchObservedRunningTime="2026-11-03 11:57:35.142833802 +0000 UTC m=+396.714417711"
Nov 03 11:57:37.649219 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:57:37.649141    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:57:46.043617 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:46.043587    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:57:53.329629 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.329586    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"]
Nov 03 11:57:53.352641 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.352614    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.388331 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.388302    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"]
Nov 03 11:57:53.433087 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.433050    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 11:57:53.433277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.433184    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:57:53.433277 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.433199    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 11:57:53.447563 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.447533    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 11:57:53.447659 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.447618    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 11:57:53.447659 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.447618    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-8xjwd\""
Nov 03 11:57:53.459018 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.458997    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5m5p\" (UniqueName: \"kubernetes.io/projected/9fd04122-4b4e-489b-ad84-48a609e191ef-kube-api-access-n5m5p\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.459076 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.459056    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fd04122-4b4e-489b-ad84-48a609e191ef-manager-config\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.459170 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.459152    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd04122-4b4e-489b-ad84-48a609e191ef-metrics-certs\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.459212 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.459188    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fd04122-4b4e-489b-ad84-48a609e191ef-cert\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.560295 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.560268    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fd04122-4b4e-489b-ad84-48a609e191ef-manager-config\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.560453 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.560309    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd04122-4b4e-489b-ad84-48a609e191ef-metrics-certs\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.560453 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.560331    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fd04122-4b4e-489b-ad84-48a609e191ef-cert\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.560453 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.560352    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5m5p\" (UniqueName: \"kubernetes.io/projected/9fd04122-4b4e-489b-ad84-48a609e191ef-kube-api-access-n5m5p\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.560936 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.560915    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fd04122-4b4e-489b-ad84-48a609e191ef-manager-config\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.563048 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.563021    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd04122-4b4e-489b-ad84-48a609e191ef-metrics-certs\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.563140 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.563033    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fd04122-4b4e-489b-ad84-48a609e191ef-cert\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.614669 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.614593    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5m5p\" (UniqueName: \"kubernetes.io/projected/9fd04122-4b4e-489b-ad84-48a609e191ef-kube-api-access-n5m5p\") pod \"jobset-controller-manager-857f98dd68-7hsrv\" (UID: \"9fd04122-4b4e-489b-ad84-48a609e191ef\") " pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:53.665693 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:53.665663    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:54.028118 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:54.028090    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"]
Nov 03 11:57:54.030649 ip-10-0-136-245 kubenswrapper[4546]: W1103 11:57:54.030617    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd04122_4b4e_489b_ad84_48a609e191ef.slice/crio-3baafa9475cb0cc8182b743121dad0307d1957ff55bf9b86aa0addbe5028528d WatchSource:0}: Error finding container 3baafa9475cb0cc8182b743121dad0307d1957ff55bf9b86aa0addbe5028528d: Status 404 returned error can't find the container with id 3baafa9475cb0cc8182b743121dad0307d1957ff55bf9b86aa0addbe5028528d
Nov 03 11:57:54.109040 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:54.109011    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv" event={"ID":"9fd04122-4b4e-489b-ad84-48a609e191ef","Type":"ContainerStarted","Data":"3baafa9475cb0cc8182b743121dad0307d1957ff55bf9b86aa0addbe5028528d"}
Nov 03 11:57:58.126515 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:58.126479    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv" event={"ID":"9fd04122-4b4e-489b-ad84-48a609e191ef","Type":"ContainerStarted","Data":"51887309ebbb4579d7b047cb313df8b82aebf348bf061561cf7e44dfa29886ad"}
Nov 03 11:57:58.126992 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:58.126581    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:57:58.234500 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:57:58.234447    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv" podStartSLOduration=1.352082991 podStartE2EDuration="5.234425157s" totalImagesPullingTime="3.882342166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:58.233064612 +0000 UTC m=+419.804648537" watchObservedRunningTime="2026-11-03 11:57:58.234425157 +0000 UTC m=+419.806009070"
Nov 03 11:58:07.650944 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:58:07.650904    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:58:09.136732 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:09.136700    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-857f98dd68-7hsrv"
Nov 03 11:58:09.889578 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:09.889548    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:58:09.889861 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:09.889822    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl" podUID="789a7eb3-47cb-4717-9739-2516383aceb5" containerName="limitador" containerID="cri-o://041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e" gracePeriod=30
Nov 03 11:58:10.829970 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:10.829944    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:58:10.912742 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:10.912705    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file\" (UniqueName: \"kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file\") pod \"789a7eb3-47cb-4717-9739-2516383aceb5\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") "
Nov 03 11:58:10.912930 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:10.912774    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc\" (UniqueName: \"kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc\") pod \"789a7eb3-47cb-4717-9739-2516383aceb5\" (UID: \"789a7eb3-47cb-4717-9739-2516383aceb5\") "
Nov 03 11:58:10.913131 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:10.913106    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file" pod "789a7eb3-47cb-4717-9739-2516383aceb5" (UID: "789a7eb3-47cb-4717-9739-2516383aceb5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:58:10.915071 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:10.915047    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc" pod "789a7eb3-47cb-4717-9739-2516383aceb5" (UID: "789a7eb3-47cb-4717-9739-2516383aceb5"). InnerVolumeSpecName "kube-api-access-5s9pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:58:11.014075 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.014039    4546 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/789a7eb3-47cb-4717-9739-2516383aceb5-config-file\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:11.014075 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.014073    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9pc\" (UniqueName: \"kubernetes.io/projected/789a7eb3-47cb-4717-9739-2516383aceb5-kube-api-access-5s9pc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:11.180185 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.180153    4546 generic.go:356] "Generic (PLEG): container finished" podID="789a7eb3-47cb-4717-9739-2516383aceb5" containerID="041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e" exitCode=0
Nov 03 11:58:11.180324 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.180201    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"
Nov 03 11:58:11.180324 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.180209    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl" event={"ID":"789a7eb3-47cb-4717-9739-2516383aceb5","Type":"ContainerDied","Data":"041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e"}
Nov 03 11:58:11.180324 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.180250    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nrqtl" event={"ID":"789a7eb3-47cb-4717-9739-2516383aceb5","Type":"ContainerDied","Data":"a3b9d9068c03a741421257806ace55ee06f86c67626d5d4f8436b7b12b9cb98c"}
Nov 03 11:58:11.180324 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.180272    4546 scope.go:122] "RemoveContainer" containerID="041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e"
Nov 03 11:58:11.193225 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.193075    4546 scope.go:122] "RemoveContainer" containerID="041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e"
Nov 03 11:58:11.193565 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:58:11.193542    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e\": container with ID starting with 041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e not found: ID does not exist" containerID="041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e"
Nov 03 11:58:11.193655 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.193575    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e"} err="failed to get container status \"041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e\": rpc error: code = NotFound desc = could not find container \"041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e\": container with ID starting with 041630827ecc9187a91a21850c160127d294a1957bbe70f8d89e3e8d7cae5c1e not found: ID does not exist"
Nov 03 11:58:11.329963 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.329883    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:58:11.375991 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:11.375956    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nrqtl"]
Nov 03 11:58:12.570970 ip-10-0-136-245 kubenswrapper[4546]: I1103 11:58:12.570934    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789a7eb3-47cb-4717-9739-2516383aceb5" path="/var/lib/kubelet/pods/789a7eb3-47cb-4717-9739-2516383aceb5/volumes"
Nov 03 11:58:37.649587 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:58:37.649550    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:59:07.648518 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:59:07.648437    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 11:59:37.649547 ip-10-0-136-245 kubenswrapper[4546]: E1103 11:59:37.649505    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:00:00.164722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.164682    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"]
Nov 03 12:00:00.165251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.165177    4546 cpu_manager.go:414] "containerMap: removing container" podUID="789a7eb3-47cb-4717-9739-2516383aceb5" containerName="limitador"
Nov 03 12:00:00.165251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.165196    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789a7eb3-47cb-4717-9739-2516383aceb5" containerName="limitador"
Nov 03 12:00:00.165366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.165294    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="789a7eb3-47cb-4717-9739-2516383aceb5" containerName="limitador"
Nov 03 12:00:00.168336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.168311    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.217368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.217337    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:00:00.263108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.263084    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:00:00.263514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.263500    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:00:00.263798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.263780    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895120-j5k98\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.263848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.263823    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rszp\" (UniqueName: \"kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp\") pod \"maas-api-key-cleanup-29895120-j5k98\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.264731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.264712    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"]
Nov 03 12:00:00.365156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.365128    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895120-j5k98\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.365294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.365166    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rszp\" (UniqueName: \"kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp\") pod \"maas-api-key-cleanup-29895120-j5k98\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.365687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.365670    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895120-j5k98\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.419442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.419385    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rszp\" (UniqueName: \"kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp\") pod \"maas-api-key-cleanup-29895120-j5k98\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.481431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.481401    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:00.843074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:00.843046    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"]
Nov 03 12:00:00.845922 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:00:00.845877    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59b119f_3f01_473c_a03a_a6311075b080.slice/crio-bd73376516151c435c157627e70d1b3ca3cd521063f4ef04d1e7d19a63842f66 WatchSource:0}: Error finding container bd73376516151c435c157627e70d1b3ca3cd521063f4ef04d1e7d19a63842f66: Status 404 returned error can't find the container with id bd73376516151c435c157627e70d1b3ca3cd521063f4ef04d1e7d19a63842f66
Nov 03 12:00:01.611182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:01.611147    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" event={"ID":"a59b119f-3f01-473c-a03a-a6311075b080","Type":"ContainerStarted","Data":"bd73376516151c435c157627e70d1b3ca3cd521063f4ef04d1e7d19a63842f66"}
Nov 03 12:00:07.649370 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:00:07.649334    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:00:14.662288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:14.662252    4546 generic.go:356] "Generic (PLEG): container finished" podID="a59b119f-3f01-473c-a03a-a6311075b080" containerID="70cdc6ccc09e2c1c7ed5eb38911995268cd3eaa0e59b2f9dd5fd24d94a76453a" exitCode=60
Nov 03 12:00:14.662705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:14.662318    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" event={"ID":"a59b119f-3f01-473c-a03a-a6311075b080","Type":"ContainerDied","Data":"70cdc6ccc09e2c1c7ed5eb38911995268cd3eaa0e59b2f9dd5fd24d94a76453a"}
Nov 03 12:00:14.662705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:14.662532    4546 scope.go:122] "RemoveContainer" containerID="70cdc6ccc09e2c1c7ed5eb38911995268cd3eaa0e59b2f9dd5fd24d94a76453a"
Nov 03 12:00:15.668227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:15.668152    4546 generic.go:356] "Generic (PLEG): container finished" podID="a59b119f-3f01-473c-a03a-a6311075b080" containerID="ff8b3e7a804f5928655ad90c7e99b0d284f016543376b69a1f24a7141cbd9fbd" exitCode=60
Nov 03 12:00:15.668227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:15.668210    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" event={"ID":"a59b119f-3f01-473c-a03a-a6311075b080","Type":"ContainerDied","Data":"ff8b3e7a804f5928655ad90c7e99b0d284f016543376b69a1f24a7141cbd9fbd"}
Nov 03 12:00:15.668680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:15.668252    4546 scope.go:122] "RemoveContainer" containerID="70cdc6ccc09e2c1c7ed5eb38911995268cd3eaa0e59b2f9dd5fd24d94a76453a"
Nov 03 12:00:15.668680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:15.668405    4546 scope.go:122] "RemoveContainer" containerID="ff8b3e7a804f5928655ad90c7e99b0d284f016543376b69a1f24a7141cbd9fbd"
Nov 03 12:00:15.668680 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:00:15.668627    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895120-j5k98_odh-ai-gateway-infra(a59b119f-3f01-473c-a03a-a6311075b080)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" podUID="a59b119f-3f01-473c-a03a-a6311075b080"
Nov 03 12:00:16.673326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:16.673298    4546 scope.go:122] "RemoveContainer" containerID="ff8b3e7a804f5928655ad90c7e99b0d284f016543376b69a1f24a7141cbd9fbd"
Nov 03 12:00:16.673715 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:00:16.673455    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895120-j5k98_odh-ai-gateway-infra(a59b119f-3f01-473c-a03a-a6311075b080)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" podUID="a59b119f-3f01-473c-a03a-a6311075b080"
Nov 03 12:00:28.567624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:28.567593    4546 scope.go:122] "RemoveContainer" containerID="ff8b3e7a804f5928655ad90c7e99b0d284f016543376b69a1f24a7141cbd9fbd"
Nov 03 12:00:29.723019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:29.722987    4546 generic.go:356] "Generic (PLEG): container finished" podID="a59b119f-3f01-473c-a03a-a6311075b080" containerID="1034479fcdd9414deaff293586fabd337f710d09c37d76e01c31b63bc0b19e1e" exitCode=60
Nov 03 12:00:29.723428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:29.723055    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" event={"ID":"a59b119f-3f01-473c-a03a-a6311075b080","Type":"ContainerDied","Data":"1034479fcdd9414deaff293586fabd337f710d09c37d76e01c31b63bc0b19e1e"}
Nov 03 12:00:29.723428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:29.723111    4546 scope.go:122] "RemoveContainer" containerID="ff8b3e7a804f5928655ad90c7e99b0d284f016543376b69a1f24a7141cbd9fbd"
Nov 03 12:00:29.723428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:29.723357    4546 scope.go:122] "RemoveContainer" containerID="1034479fcdd9414deaff293586fabd337f710d09c37d76e01c31b63bc0b19e1e"
Nov 03 12:00:29.723587 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:00:29.723567    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895120-j5k98_odh-ai-gateway-infra(a59b119f-3f01-473c-a03a-a6311075b080)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" podUID="a59b119f-3f01-473c-a03a-a6311075b080"
Nov 03 12:00:30.838718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:30.838686    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"]
Nov 03 12:00:30.963798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:30.963778    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:31.010681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.010655    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp\" (UniqueName: \"kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp\") pod \"a59b119f-3f01-473c-a03a-a6311075b080\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") "
Nov 03 12:00:31.010865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.010722    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca\") pod \"a59b119f-3f01-473c-a03a-a6311075b080\" (UID: \"a59b119f-3f01-473c-a03a-a6311075b080\") "
Nov 03 12:00:31.011001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.010985    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca" pod "a59b119f-3f01-473c-a03a-a6311075b080" (UID: "a59b119f-3f01-473c-a03a-a6311075b080"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:00:31.012847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.012823    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp" pod "a59b119f-3f01-473c-a03a-a6311075b080" (UID: "a59b119f-3f01-473c-a03a-a6311075b080"). InnerVolumeSpecName "kube-api-access-6rszp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:00:31.111340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.111257    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59b119f-3f01-473c-a03a-a6311075b080-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:00:31.111340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.111284    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rszp\" (UniqueName: \"kubernetes.io/projected/a59b119f-3f01-473c-a03a-a6311075b080-kube-api-access-6rszp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:00:31.733134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.733107    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"
Nov 03 12:00:31.733304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.733130    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98" event={"ID":"a59b119f-3f01-473c-a03a-a6311075b080","Type":"ContainerDied","Data":"bd73376516151c435c157627e70d1b3ca3cd521063f4ef04d1e7d19a63842f66"}
Nov 03 12:00:31.733304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.733186    4546 scope.go:122] "RemoveContainer" containerID="1034479fcdd9414deaff293586fabd337f710d09c37d76e01c31b63bc0b19e1e"
Nov 03 12:00:31.884263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.884233    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"]
Nov 03 12:00:31.889215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:31.889194    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895120-j5k98"]
Nov 03 12:00:32.570050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:32.570020    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59b119f-3f01-473c-a03a-a6311075b080" path="/var/lib/kubelet/pods/a59b119f-3f01-473c-a03a-a6311075b080/volumes"
Nov 03 12:00:37.649288 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:00:37.649207    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:00:46.648062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648029    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-p7c22"]
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648304    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648314    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648326    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648331    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648341    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648346    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648420    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648428    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.650459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.648435    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59b119f-3f01-473c-a03a-a6311075b080" containerName="cleanup"
Nov 03 12:00:46.651345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.651325    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.724586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.724554    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlvt\" (UniqueName: \"kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt\") pod \"ai-gateway-controller-5d8dddb676-p7c22\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.724697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.724589    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-p7c22\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.746562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.746537    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-m9p4h\""
Nov 03 12:00:46.746562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.746547    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 12:00:46.746812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.746794    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 12:00:46.746892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.746816    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 12:00:46.746892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.746860    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 12:00:46.748187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.748168    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-p7c22"]
Nov 03 12:00:46.825488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.825467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-p7c22\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.825611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.825536    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlvt\" (UniqueName: \"kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt\") pod \"ai-gateway-controller-5d8dddb676-p7c22\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.827978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.827957    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-p7c22\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.878325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.878299    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlvt\" (UniqueName: \"kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt\") pod \"ai-gateway-controller-5d8dddb676-p7c22\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:46.963388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:46.963338    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:47.327100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:47.327068    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-p7c22"]
Nov 03 12:00:47.330548 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:00:47.330522    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02304a22_f650_4eb1_8009_ac102976cdff.slice/crio-a78219978492b7747ec139c8eb27f7fcefe4140cf89814f2c84db354d5c35d3b WatchSource:0}: Error finding container a78219978492b7747ec139c8eb27f7fcefe4140cf89814f2c84db354d5c35d3b: Status 404 returned error can't find the container with id a78219978492b7747ec139c8eb27f7fcefe4140cf89814f2c84db354d5c35d3b
Nov 03 12:00:47.793541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:47.793512    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22" event={"ID":"02304a22-f650-4eb1-8009-ac102976cdff","Type":"ContainerStarted","Data":"a78219978492b7747ec139c8eb27f7fcefe4140cf89814f2c84db354d5c35d3b"}
Nov 03 12:00:52.815551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:52.815514    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22" event={"ID":"02304a22-f650-4eb1-8009-ac102976cdff","Type":"ContainerStarted","Data":"46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281"}
Nov 03 12:00:52.815992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:52.815658    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:00:52.923279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:52.923233    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22" podStartSLOduration=1.791196298 podStartE2EDuration="6.92322097s" totalImagesPullingTime="5.132024672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:00:52.92281825 +0000 UTC m=+594.494402185" watchObservedRunningTime="2026-11-03 12:00:52.92322097 +0000 UTC m=+594.494804882"
Nov 03 12:00:55.574210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.574174    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"]
Nov 03 12:00:55.577536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.577514    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.672122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.672085    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 12:00:55.672122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.672089    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xs659\""
Nov 03 12:00:55.673392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.673373    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"]
Nov 03 12:00:55.705847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.705813    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kmmh4\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.705847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.705843    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkrh\" (UniqueName: \"kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh\") pod \"payload-pre-processing-7b98bb678c-kmmh4\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.729949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.729916    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"]
Nov 03 12:00:55.733577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.733559    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:55.787133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.787103    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"]
Nov 03 12:00:55.807104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.807076    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kmmh4\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.807104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.807107    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkrh\" (UniqueName: \"kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh\") pod \"payload-pre-processing-7b98bb678c-kmmh4\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.807835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.807815    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kmmh4\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.861204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.861140    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkrh\" (UniqueName: \"kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh\") pod \"payload-pre-processing-7b98bb678c-kmmh4\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.890037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.890016    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:55.908277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.908254    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgf98\" (UniqueName: \"kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98\") pod \"payload-processing-55fbbbf9dc-zmp2m\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:55.908364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:55.908303    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-zmp2m\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:56.009540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.009513    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-zmp2m\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:56.009702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.009580    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf98\" (UniqueName: \"kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98\") pod \"payload-processing-55fbbbf9dc-zmp2m\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:56.010214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.010192    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-zmp2m\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:56.063668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.063641    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgf98\" (UniqueName: \"kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98\") pod \"payload-processing-55fbbbf9dc-zmp2m\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:56.253852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.253716    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"]
Nov 03 12:00:56.256766 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:00:56.256727    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ff9dfe_4265_4629_9aba_b7cb5ccd539f.slice/crio-9dc26df55dd449601a50ebc628a0cec791a57fadbc49e95bc5490505dadcbe5f WatchSource:0}: Error finding container 9dc26df55dd449601a50ebc628a0cec791a57fadbc49e95bc5490505dadcbe5f: Status 404 returned error can't find the container with id 9dc26df55dd449601a50ebc628a0cec791a57fadbc49e95bc5490505dadcbe5f
Nov 03 12:00:56.345465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.345435    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:56.705214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.705149    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"]
Nov 03 12:00:56.708021 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:00:56.707988    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10c10e5_e265_4bcf_b38a_1de049716594.slice/crio-a2924f7ada07d659b56664c5c747731f366550ff0632ed6c287cdfffca5618f5 WatchSource:0}: Error finding container a2924f7ada07d659b56664c5c747731f366550ff0632ed6c287cdfffca5618f5: Status 404 returned error can't find the container with id a2924f7ada07d659b56664c5c747731f366550ff0632ed6c287cdfffca5618f5
Nov 03 12:00:56.831960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.831928    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4" event={"ID":"58ff9dfe-4265-4629-9aba-b7cb5ccd539f","Type":"ContainerStarted","Data":"9dc26df55dd449601a50ebc628a0cec791a57fadbc49e95bc5490505dadcbe5f"}
Nov 03 12:00:56.833087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:56.833067    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m" event={"ID":"b10c10e5-e265-4bcf-b38a-1de049716594","Type":"ContainerStarted","Data":"a2924f7ada07d659b56664c5c747731f366550ff0632ed6c287cdfffca5618f5"}
Nov 03 12:00:59.850249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:59.850215    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4" event={"ID":"58ff9dfe-4265-4629-9aba-b7cb5ccd539f","Type":"ContainerStarted","Data":"d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2"}
Nov 03 12:00:59.850667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:59.850285    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:00:59.851802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:59.851779    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m" event={"ID":"b10c10e5-e265-4bcf-b38a-1de049716594","Type":"ContainerStarted","Data":"34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39"}
Nov 03 12:00:59.851911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:59.851883    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:00:59.956845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:00:59.956734    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4" podStartSLOduration=1.838556535 podStartE2EDuration="4.956718242s" totalImagesPullingTime="3.118161707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:00:59.955885993 +0000 UTC m=+601.527469907" watchObservedRunningTime="2026-11-03 12:00:59.956718242 +0000 UTC m=+601.528302154"
Nov 03 12:01:00.063027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:00.062971    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m" podStartSLOduration=2.390084047 podStartE2EDuration="5.062952022s" totalImagesPullingTime="2.672867975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:00.061239026 +0000 UTC m=+601.632822948" watchObservedRunningTime="2026-11-03 12:01:00.062952022 +0000 UTC m=+601.634535934"
Nov 03 12:01:03.822492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:03.822460    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:01:04.169737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.169659    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"]
Nov 03 12:01:04.173224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.173206    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.228535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.228501    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"]
Nov 03 12:01:04.281632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.281602    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3e369-07e6-4332-a68a-58c77461df97-metrics-certs\") pod \"ai-gateway-controller-6cd79d8544-f6nxm\" (UID: \"abc3e369-07e6-4332-a68a-58c77461df97\") " pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.281632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.281631    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4xz\" (UniqueName: \"kubernetes.io/projected/abc3e369-07e6-4332-a68a-58c77461df97-kube-api-access-zv4xz\") pod \"ai-gateway-controller-6cd79d8544-f6nxm\" (UID: \"abc3e369-07e6-4332-a68a-58c77461df97\") " pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.383028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.382986    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3e369-07e6-4332-a68a-58c77461df97-metrics-certs\") pod \"ai-gateway-controller-6cd79d8544-f6nxm\" (UID: \"abc3e369-07e6-4332-a68a-58c77461df97\") " pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.383197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.383034    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4xz\" (UniqueName: \"kubernetes.io/projected/abc3e369-07e6-4332-a68a-58c77461df97-kube-api-access-zv4xz\") pod \"ai-gateway-controller-6cd79d8544-f6nxm\" (UID: \"abc3e369-07e6-4332-a68a-58c77461df97\") " pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.385570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.385548    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3e369-07e6-4332-a68a-58c77461df97-metrics-certs\") pod \"ai-gateway-controller-6cd79d8544-f6nxm\" (UID: \"abc3e369-07e6-4332-a68a-58c77461df97\") " pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.436261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.436195    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4xz\" (UniqueName: \"kubernetes.io/projected/abc3e369-07e6-4332-a68a-58c77461df97-kube-api-access-zv4xz\") pod \"ai-gateway-controller-6cd79d8544-f6nxm\" (UID: \"abc3e369-07e6-4332-a68a-58c77461df97\") " pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.485037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.485011    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:04.846354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.846331    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"]
Nov 03 12:01:04.848600 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:01:04.848556    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc3e369_07e6_4332_a68a_58c77461df97.slice/crio-122e814e43e3b62946f8f0dd4266cb69b81174873138010e82693d8044cc606c WatchSource:0}: Error finding container 122e814e43e3b62946f8f0dd4266cb69b81174873138010e82693d8044cc606c: Status 404 returned error can't find the container with id 122e814e43e3b62946f8f0dd4266cb69b81174873138010e82693d8044cc606c
Nov 03 12:01:04.872244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:04.872217    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm" event={"ID":"abc3e369-07e6-4332-a68a-58c77461df97","Type":"ContainerStarted","Data":"122e814e43e3b62946f8f0dd4266cb69b81174873138010e82693d8044cc606c"}
Nov 03 12:01:05.879619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:05.879585    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm" event={"ID":"abc3e369-07e6-4332-a68a-58c77461df97","Type":"ContainerStarted","Data":"c4777378ef68ee2713afbe82e71fccb4732ac0cce46760b9d392dd14a746819c"}
Nov 03 12:01:05.880018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:05.879675    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:05.986887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:05.986744    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm" podStartSLOduration=1.181014953 podStartE2EDuration="1.986727015s" totalImagesPullingTime="805.712062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:05.985175934 +0000 UTC m=+607.556759838" watchObservedRunningTime="2026-11-03 12:01:05.986727015 +0000 UTC m=+607.558310929"
Nov 03 12:01:07.650489 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:01:07.650454    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:01:10.858723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:10.858692    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 12:01:10.859200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:10.858772    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 12:01:16.886816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:16.886785    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cd79d8544-f6nxm"
Nov 03 12:01:17.040927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.040896    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-p7c22"]
Nov 03 12:01:17.041113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.041093    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22" podUID="02304a22-f650-4eb1-8009-ac102976cdff" containerName="manager" containerID="cri-o://46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281" gracePeriod=30
Nov 03 12:01:17.280394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.280370    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:01:17.368643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.368607    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs\") pod \"02304a22-f650-4eb1-8009-ac102976cdff\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") "
Nov 03 12:01:17.368643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.368646    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt\" (UniqueName: \"kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt\") pod \"02304a22-f650-4eb1-8009-ac102976cdff\" (UID: \"02304a22-f650-4eb1-8009-ac102976cdff\") "
Nov 03 12:01:17.371060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.371023    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt" pod "02304a22-f650-4eb1-8009-ac102976cdff" (UID: "02304a22-f650-4eb1-8009-ac102976cdff"). InnerVolumeSpecName "kube-api-access-ddlvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:01:17.371171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.371030    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs" pod "02304a22-f650-4eb1-8009-ac102976cdff" (UID: "02304a22-f650-4eb1-8009-ac102976cdff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:01:17.469421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.469385    4546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02304a22-f650-4eb1-8009-ac102976cdff-metrics-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:01:17.469421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.469420    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlvt\" (UniqueName: \"kubernetes.io/projected/02304a22-f650-4eb1-8009-ac102976cdff-kube-api-access-ddlvt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:01:17.928993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.928913    4546 generic.go:356] "Generic (PLEG): container finished" podID="02304a22-f650-4eb1-8009-ac102976cdff" containerID="46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281" exitCode=0
Nov 03 12:01:17.928993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.928958    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22"
Nov 03 12:01:17.929410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.928998    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22" event={"ID":"02304a22-f650-4eb1-8009-ac102976cdff","Type":"ContainerDied","Data":"46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281"}
Nov 03 12:01:17.929410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.929054    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-p7c22" event={"ID":"02304a22-f650-4eb1-8009-ac102976cdff","Type":"ContainerDied","Data":"a78219978492b7747ec139c8eb27f7fcefe4140cf89814f2c84db354d5c35d3b"}
Nov 03 12:01:17.929410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.929079    4546 scope.go:122] "RemoveContainer" containerID="46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281"
Nov 03 12:01:17.938036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.938019    4546 scope.go:122] "RemoveContainer" containerID="46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281"
Nov 03 12:01:17.938282 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:01:17.938267    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281\": container with ID starting with 46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281 not found: ID does not exist" containerID="46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281"
Nov 03 12:01:17.938327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:17.938291    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281"} err="failed to get container status \"46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281\": rpc error: code = NotFound desc = could not find container \"46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281\": container with ID starting with 46fafbdaaea6efef4d87d12de9a3d10e401d6f561dddc1bd15cacb2f47c93281 not found: ID does not exist"
Nov 03 12:01:18.080421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:18.080394    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-p7c22"]
Nov 03 12:01:18.126342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:18.126316    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-p7c22"]
Nov 03 12:01:18.572195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:18.572163    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02304a22-f650-4eb1-8009-ac102976cdff" path="/var/lib/kubelet/pods/02304a22-f650-4eb1-8009-ac102976cdff/volumes"
Nov 03 12:01:28.188213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.188180    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"]
Nov 03 12:01:28.188628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.188523    4546 cpu_manager.go:414] "containerMap: removing container" podUID="02304a22-f650-4eb1-8009-ac102976cdff" containerName="manager"
Nov 03 12:01:28.188628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.188538    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02304a22-f650-4eb1-8009-ac102976cdff" containerName="manager"
Nov 03 12:01:28.188628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.188611    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="02304a22-f650-4eb1-8009-ac102976cdff" containerName="manager"
Nov 03 12:01:28.195071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.195048    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.246104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.246042    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2n5\" (UniqueName: \"kubernetes.io/projected/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-kube-api-access-5s2n5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.246628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.246583    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.246794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.246668    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.246867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.246815    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.247249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.247052    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.247249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.247098    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.251185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.250005    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"]
Nov 03 12:01:28.255583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.254982    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 12:01:28.292205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.292155    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 12:01:28.292205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.292184    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 12:01:28.292415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.292242    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8vc6k\""
Nov 03 12:01:28.348537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.348499    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.348715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.348548    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2n5\" (UniqueName: \"kubernetes.io/projected/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-kube-api-access-5s2n5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.348715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.348578    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.348715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.348604    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.348715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.348644    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.348715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.348693    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.349098 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.349072    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.349253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.349129    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.349253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.349148    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.351169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.351146    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.351253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.351151    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.402614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.402583    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2n5\" (UniqueName: \"kubernetes.io/projected/bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c-kube-api-access-5s2n5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c\" (UID: \"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.506838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.506801    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:28.871657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.871574    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"]
Nov 03 12:01:28.874621 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:01:28.874594    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4799b4_b7ce_4a2b_bc17_8b54823b2e9c.slice/crio-085d8a450f093f9f9d4b0ba150bd9de96e87770450d6aaa5bfc78c824d93da04 WatchSource:0}: Error finding container 085d8a450f093f9f9d4b0ba150bd9de96e87770450d6aaa5bfc78c824d93da04: Status 404 returned error can't find the container with id 085d8a450f093f9f9d4b0ba150bd9de96e87770450d6aaa5bfc78c824d93da04
Nov 03 12:01:28.973795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:28.973740    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c" event={"ID":"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c","Type":"ContainerStarted","Data":"085d8a450f093f9f9d4b0ba150bd9de96e87770450d6aaa5bfc78c824d93da04"}
Nov 03 12:01:31.988838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:31.988783    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c" event={"ID":"bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c","Type":"ContainerStarted","Data":"3f3b964b0d294a7c52f22b8a43327c2616dd85b3cd64808a26eb0758e18987bb"}
Nov 03 12:01:32.097455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:32.097390    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c" podStartSLOduration=1.872588693 podStartE2EDuration="4.097372088s" totalImagesPullingTime="2.224783395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:32.096570936 +0000 UTC m=+633.668154845" watchObservedRunningTime="2026-11-03 12:01:32.097372088 +0000 UTC m=+633.668956001"
Nov 03 12:01:32.507387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:32.507357    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:37.650662 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:01:37.650622    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:01:38.507573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:38.507527    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:38.520667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:38.520638    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:39.028763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:39.028714    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c"
Nov 03 12:01:40.237512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.237482    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"]
Nov 03 12:01:40.240864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.240843    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.335391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.335359    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 12:01:40.337200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.337176    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"]
Nov 03 12:01:40.350649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.350618    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.350805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.350681    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.350870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.350851    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.350908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.350884    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.350958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.350912    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd62\" (UniqueName: \"kubernetes.io/projected/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-kube-api-access-kdd62\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.351048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.351026    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452044    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452077    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452094    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd62\" (UniqueName: \"kubernetes.io/projected/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-kube-api-access-kdd62\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452125    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452179    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452206    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452423    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452565    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.452592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.452591    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.454616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.454590    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.454811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.454791    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.505248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.505184    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd62\" (UniqueName: \"kubernetes.io/projected/5b0a55f3-2561-489e-aa3e-ebeb24ca66b4-kube-api-access-kdd62\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kpcdf\" (UID: \"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.553373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.553343    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:40.913683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:40.913630    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"]
Nov 03 12:01:40.915802 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:01:40.915774    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0a55f3_2561_489e_aa3e_ebeb24ca66b4.slice/crio-04af261ad6b526b1291a252dde10df11392f1a30665bd2748aebc4801d6346ee WatchSource:0}: Error finding container 04af261ad6b526b1291a252dde10df11392f1a30665bd2748aebc4801d6346ee: Status 404 returned error can't find the container with id 04af261ad6b526b1291a252dde10df11392f1a30665bd2748aebc4801d6346ee
Nov 03 12:01:41.026548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:41.026520    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf" event={"ID":"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4","Type":"ContainerStarted","Data":"04af261ad6b526b1291a252dde10df11392f1a30665bd2748aebc4801d6346ee"}
Nov 03 12:01:42.032375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:42.032339    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf" event={"ID":"5b0a55f3-2561-489e-aa3e-ebeb24ca66b4","Type":"ContainerStarted","Data":"228cd981d951b48529defba47f2b36d8fb831e836217a1c4820ed40ee4ae2b60"}
Nov 03 12:01:42.139968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:42.139926    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf" podStartSLOduration=1.514912612 podStartE2EDuration="2.139912424s" totalImagesPullingTime="624.999812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:42.139253552 +0000 UTC m=+643.710837465" watchObservedRunningTime="2026-11-03 12:01:42.139912424 +0000 UTC m=+643.711496376"
Nov 03 12:01:42.554440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:42.554408    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:47.598606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.598572    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"]
Nov 03 12:01:47.602147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.602126    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.656134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.656105    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 12:01:47.700050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.700018    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg57t\""
Nov 03 12:01:47.702626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.702602    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"]
Nov 03 12:01:47.712420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712396    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712428    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbw4k\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712455    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712471    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712576    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712601    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712615    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712671    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.712779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.712708    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813613    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813660    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813678    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813694    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813731    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813794    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.813852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813850    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813873    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbw4k\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.813914    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.814162    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.814252    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.814332    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.814452    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.814625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.814466    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.816155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.816136    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.816365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.816347    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.869485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.869412    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.869485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.869476    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbw4k\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ltzvx\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:47.914218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:47.914186    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:48.323326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:48.323291    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"]
Nov 03 12:01:48.327569 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:01:48.327536    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6951b33a_2fdd_4df7_9f8e_339c2a2c1b4c.slice/crio-65c5628dc28bb66c0b359546ad6039d89a4c59a80006ec004c258b1d9b2bfc6e WatchSource:0}: Error finding container 65c5628dc28bb66c0b359546ad6039d89a4c59a80006ec004c258b1d9b2bfc6e: Status 404 returned error can't find the container with id 65c5628dc28bb66c0b359546ad6039d89a4c59a80006ec004c258b1d9b2bfc6e
Nov 03 12:01:48.329913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:48.329894    4546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:01:49.066324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:49.066290    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" event={"ID":"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c","Type":"ContainerStarted","Data":"65c5628dc28bb66c0b359546ad6039d89a4c59a80006ec004c258b1d9b2bfc6e"}
Nov 03 12:01:50.553845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.553807    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:50.559667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.559643    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"]
Nov 03 12:01:50.563474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.563453    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.573261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.573208    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:50.614666 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.614639    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 12:01:50.641202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.641167    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.641345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.641238    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.641345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.641276    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06504b-9351-4e76-971b-31668db4bb5c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.641473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.641402    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.641540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.641520    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.641585 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.641560    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8c26\" (UniqueName: \"kubernetes.io/projected/bc06504b-9351-4e76-971b-31668db4bb5c-kube-api-access-q8c26\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.661195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.661168    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"]
Nov 03 12:01:50.742949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.742914    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06504b-9351-4e76-971b-31668db4bb5c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.742999    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743050    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743080    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8c26\" (UniqueName: \"kubernetes.io/projected/bc06504b-9351-4e76-971b-31668db4bb5c-kube-api-access-q8c26\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743129    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743161    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743871 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743437    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.743871 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743707    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.744082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.743909    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.745985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.745953    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc06504b-9351-4e76-971b-31668db4bb5c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.748437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.748417    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc06504b-9351-4e76-971b-31668db4bb5c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.798890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.798860    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8c26\" (UniqueName: \"kubernetes.io/projected/bc06504b-9351-4e76-971b-31668db4bb5c-kube-api-access-q8c26\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74\" (UID: \"bc06504b-9351-4e76-971b-31668db4bb5c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:50.880740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:50.880675    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:51.092743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:51.092714    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf"
Nov 03 12:01:51.256652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:51.256633    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"]
Nov 03 12:01:51.259043 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:01:51.259014    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc06504b_9351_4e76_971b_31668db4bb5c.slice/crio-aa1931623570b0cc3d6233a850a7b75b130d00eb9e2ccdd42ff3d96be0e0b26b WatchSource:0}: Error finding container aa1931623570b0cc3d6233a850a7b75b130d00eb9e2ccdd42ff3d96be0e0b26b: Status 404 returned error can't find the container with id aa1931623570b0cc3d6233a850a7b75b130d00eb9e2ccdd42ff3d96be0e0b26b
Nov 03 12:01:52.083396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.083356    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74" event={"ID":"bc06504b-9351-4e76-971b-31668db4bb5c","Type":"ContainerStarted","Data":"c21761259d811e5a23de26295d4f7dcec0c7d7277cfd1e9b0a0a95ff8d414e91"}
Nov 03 12:01:52.083885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.083412    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74" event={"ID":"bc06504b-9351-4e76-971b-31668db4bb5c","Type":"ContainerStarted","Data":"aa1931623570b0cc3d6233a850a7b75b130d00eb9e2ccdd42ff3d96be0e0b26b"}
Nov 03 12:01:52.192258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.192140    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74" podStartSLOduration=1.5487913899999999 podStartE2EDuration="2.192124987s" totalImagesPullingTime="643.333597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:52.19021919 +0000 UTC m=+653.761803141" watchObservedRunningTime="2026-11-03 12:01:52.192124987 +0000 UTC m=+653.763708898"
Nov 03 12:01:52.464580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.464554    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:01:52.464650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.464618    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:01:52.464650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.464645    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:01:52.881563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:52.881529    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:01:53.089402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.089368    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" event={"ID":"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c","Type":"ContainerStarted","Data":"e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5"}
Nov 03 12:01:53.245997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.245947    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podStartSLOduration=2.112220533 podStartE2EDuration="6.245932966s" totalImagesPullingTime="4.133712433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:53.245234284 +0000 UTC m=+654.816818256" watchObservedRunningTime="2026-11-03 12:01:53.245932966 +0000 UTC m=+654.817516878"
Nov 03 12:01:53.931788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.928805    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:01:53.931982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.931882    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 12:01:53.931982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.931940    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 12:01:53.968818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.968782    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"]
Nov 03 12:01:53.972357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:53.972338    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.033786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.030203    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 12:01:54.068807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.068777    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"]
Nov 03 12:01:54.074459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.074436    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e31e600-1afa-400e-8acb-5c32a3c2179d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.074542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.074475    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcx5\" (UniqueName: \"kubernetes.io/projected/9e31e600-1afa-400e-8acb-5c32a3c2179d-kube-api-access-rtcx5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.074542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.074495    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.074614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.074558    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.074676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.074662    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.074715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.074685    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.175982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.175952    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e31e600-1afa-400e-8acb-5c32a3c2179d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.175995    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcx5\" (UniqueName: \"kubernetes.io/projected/9e31e600-1afa-400e-8acb-5c32a3c2179d-kube-api-access-rtcx5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176018    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176061    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176388    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176440    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176478    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176539    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.176898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.176837    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.178538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.178517    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e31e600-1afa-400e-8acb-5c32a3c2179d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.178538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.178530    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e31e600-1afa-400e-8acb-5c32a3c2179d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.232330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.232302    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcx5\" (UniqueName: \"kubernetes.io/projected/9e31e600-1afa-400e-8acb-5c32a3c2179d-kube-api-access-rtcx5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt\" (UID: \"9e31e600-1afa-400e-8acb-5c32a3c2179d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.282938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.282914    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:54.677058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.676984    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"]
Nov 03 12:01:54.680467 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:01:54.680426    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e31e600_1afa_400e_8acb_5c32a3c2179d.slice/crio-43a64fc91dcc90e6bf2754c711ce861bfd45b1193658d9b1eacc4debb8c98724 WatchSource:0}: Error finding container 43a64fc91dcc90e6bf2754c711ce861bfd45b1193658d9b1eacc4debb8c98724: Status 404 returned error can't find the container with id 43a64fc91dcc90e6bf2754c711ce861bfd45b1193658d9b1eacc4debb8c98724
Nov 03 12:01:54.928834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.923217    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 12:01:54.928834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:54.923286    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 12:01:55.098378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:55.098341    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" event={"ID":"9e31e600-1afa-400e-8acb-5c32a3c2179d","Type":"ContainerStarted","Data":"43a64fc91dcc90e6bf2754c711ce861bfd45b1193658d9b1eacc4debb8c98724"}
Nov 03 12:01:55.927775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:55.922142    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 12:01:55.927775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:55.922209    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 12:01:56.103886 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:56.103853    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" event={"ID":"9e31e600-1afa-400e-8acb-5c32a3c2179d","Type":"ContainerStarted","Data":"53ea77b9978356a4b9fd7423942aa48fce9b4eead6e7bd001ef8da3e4f7be67a"}
Nov 03 12:01:56.283077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:56.283040    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:01:56.923188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:56.923147    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 12:01:56.923484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:56.923456    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 12:01:57.927773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:57.916131    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 12:01:57.927773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:57.916212    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 12:01:58.915595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:58.915520    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 12:01:58.915595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:01:58.915572    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 12:02:00.311087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:00.311054    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:02:00.311630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:00.311599    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:02:00.312461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:00.312443    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:02:00.467356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:00.467302    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podStartSLOduration=6.792193372 podStartE2EDuration="7.467284817s" totalImagesPullingTime="675.091445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:01:56.216730921 +0000 UTC m=+657.788314865" watchObservedRunningTime="2026-11-03 12:02:00.467284817 +0000 UTC m=+662.038868733"
Nov 03 12:02:00.881330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:00.881297    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:02:00.893946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:00.893915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:02:01.162495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:01.162414    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74"
Nov 03 12:02:04.283068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:04.283036    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:02:04.295322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:04.295301    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:02:05.181293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:05.181264    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:02:07.649512 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:02:07.649426    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:02:17.839114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.839082    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"]
Nov 03 12:02:17.842812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.842790    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915512    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915548    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915582    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915654    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915691    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915709    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.915831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915818    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lrk\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.916018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915847    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.916018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.915864    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:17.941732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:17.941700    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"]
Nov 03 12:02:18.016308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016273    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016310    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016330    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016457    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016583    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lrk\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016614    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016640    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016730    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016790    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.016840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016805    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.017024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.016969    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.017097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.017072    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.017356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.017330    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.017490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.017462    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.019057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.019031    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.019310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.019294    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.069843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.069813    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.070171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.070148    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lrk\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jb2gq\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.155027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.154948    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:18.570852 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:02:18.570806    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07785463_7365_4b0b_9ece_ebefa51c7469.slice/crio-bf71317af60d872288b1e31d90493d46a0a6cbb188554aeae20c5f68a3fc9ba5 WatchSource:0}: Error finding container bf71317af60d872288b1e31d90493d46a0a6cbb188554aeae20c5f68a3fc9ba5: Status 404 returned error can't find the container with id bf71317af60d872288b1e31d90493d46a0a6cbb188554aeae20c5f68a3fc9ba5
Nov 03 12:02:18.572427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.572404    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"]
Nov 03 12:02:18.572889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.572850    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:02:18.572990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.572938    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:02:18.572990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:18.572982    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:02:19.238100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:19.238060    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" event={"ID":"07785463-7365-4b0b-9ece-ebefa51c7469","Type":"ContainerStarted","Data":"d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652"}
Nov 03 12:02:19.238581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:19.238160    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" event={"ID":"07785463-7365-4b0b-9ece-ebefa51c7469","Type":"ContainerStarted","Data":"bf71317af60d872288b1e31d90493d46a0a6cbb188554aeae20c5f68a3fc9ba5"}
Nov 03 12:02:19.445376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:19.445320    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podStartSLOduration=2.44530643 podStartE2EDuration="2.44530643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:02:19.444081981 +0000 UTC m=+681.015665904" watchObservedRunningTime="2026-11-03 12:02:19.44530643 +0000 UTC m=+681.016890343"
Nov 03 12:02:20.156089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:20.156055    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:20.157644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:20.157611    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jb2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 12:02:20.157782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:20.157661    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 12:02:21.155581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:21.155543    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jb2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 12:02:21.155985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:21.155608    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 12:02:22.155827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:22.155797    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jb2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 12:02:22.156258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:22.155848    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 12:02:23.156185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:23.156151    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jb2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 12:02:23.156574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:23.156202    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 12:02:24.155736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:24.155703    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jb2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 12:02:24.155935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:24.155783    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 12:02:25.155838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:25.155801    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jb2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 12:02:25.156366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:25.155859    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 12:02:26.815611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:26.815577    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:26.816075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:26.815982    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:26.816891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:02:26.816871    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:02:37.650337 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:02:37.650296    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:03:06.604069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.604041    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"]
Nov 03 12:03:06.607935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.607915    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.698706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.698674    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"]
Nov 03 12:03:06.742736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.742703    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f732077f-openshift-default-dockercfg-9ld5s\""
Nov 03 12:03:06.759105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759072    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759116    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb67d\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759146    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759208    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759301    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759369    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759430    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759501    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.759564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.759524    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860548    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860590    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb67d\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860611    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860634    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860653    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860674    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860724    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860780    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.860954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.860803    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.861305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.861049    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.861305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.861126    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.861305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.861159    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.861453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.861393    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.861544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.861525    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.863143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.863121    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.863400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.863381    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.934643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.934610    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:06.934879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:06.934853    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb67d\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d\") pod \"e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") " pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:07.220584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:07.220553    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:07.656644 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:03:07.656562    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:03:07.689198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:07.689148    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"]
Nov 03 12:03:07.691669 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:07.691636    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode163e1eb_3878_4636_8feb_9d89f1d7ce8d.slice/crio-143a86a5582c19295756d14a42f01498e97f719f64319e02cc419257b27081f4 WatchSource:0}: Error finding container 143a86a5582c19295756d14a42f01498e97f719f64319e02cc419257b27081f4: Status 404 returned error can't find the container with id 143a86a5582c19295756d14a42f01498e97f719f64319e02cc419257b27081f4
Nov 03 12:03:07.693962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:07.693931    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:07.694036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:07.694003    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:07.694036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:07.694030    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:08.455574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.455536    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" event={"ID":"e163e1eb-3878-4636-8feb-9d89f1d7ce8d","Type":"ContainerStarted","Data":"3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"}
Nov 03 12:03:08.455834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.455601    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" event={"ID":"e163e1eb-3878-4636-8feb-9d89f1d7ce8d","Type":"ContainerStarted","Data":"143a86a5582c19295756d14a42f01498e97f719f64319e02cc419257b27081f4"}
Nov 03 12:03:08.565000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.564888    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podStartSLOduration=2.564868282 podStartE2EDuration="2.564868282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:08.563453807 +0000 UTC m=+730.135037722" watchObservedRunningTime="2026-11-03 12:03:08.564868282 +0000 UTC m=+730.136452196"
Nov 03 12:03:08.877233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.877201    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"]
Nov 03 12:03:08.880938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.880917    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.929591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.929555    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4b9370d1-openshift-default-dockercfg-j6xvp\""
Nov 03 12:03:08.939889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.939855    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"]
Nov 03 12:03:08.978777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.978726    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.978950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.978843    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.978950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.978887    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.978950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.978905    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.978950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.978922    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.978950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.978944    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.979145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.979086    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.979145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.979109    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:08.979241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:08.979155    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dgc\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080008    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080058    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080080    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080104    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080122    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080142    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080185    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080215    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080282    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dgc\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080475    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080484    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080882    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.080997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.080888    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.081277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.081253    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.082791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.082770    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.082875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.082855    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.133867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.133794    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dgc\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.133867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.133803    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token\") pod \"e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") " pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.194865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.194827    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:09.221380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.221349    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:09.226923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.226876    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:09.460197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.460124    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:09.461198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.461173    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:09.607313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.607278    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"]
Nov 03 12:03:09.620305 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:09.620278    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6997879_6745_4d61_b4f8_12f5e9420d58.slice/crio-1650af3bcd76c0b0de46f3cc89dfa0e05d643ba139fa10bfda531108b9902b7d WatchSource:0}: Error finding container 1650af3bcd76c0b0de46f3cc89dfa0e05d643ba139fa10bfda531108b9902b7d: Status 404 returned error can't find the container with id 1650af3bcd76c0b0de46f3cc89dfa0e05d643ba139fa10bfda531108b9902b7d
Nov 03 12:03:09.622621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.622579    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:09.622723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.622669    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:09.622723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:09.622714    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:10.466114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:10.466079    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" event={"ID":"c6997879-6745-4d61-b4f8-12f5e9420d58","Type":"ContainerStarted","Data":"47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"}
Nov 03 12:03:10.466595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:10.466136    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" event={"ID":"c6997879-6745-4d61-b4f8-12f5e9420d58","Type":"ContainerStarted","Data":"1650af3bcd76c0b0de46f3cc89dfa0e05d643ba139fa10bfda531108b9902b7d"}
Nov 03 12:03:10.582817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:10.582729    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podStartSLOduration=2.582710159 podStartE2EDuration="2.582710159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:10.582079449 +0000 UTC m=+732.153663422" watchObservedRunningTime="2026-11-03 12:03:10.582710159 +0000 UTC m=+732.154294072"
Nov 03 12:03:11.195179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:11.195148    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:11.200337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:11.200312    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:11.471456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:11.471426    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:11.472515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:11.472496    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:13.596744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.596704    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"]
Nov 03 12:03:13.600454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.600434    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.650545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.650511    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9ab28c7a-openshift-default-dockercfg-vz74g\""
Nov 03 12:03:13.697920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.697881    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"]
Nov 03 12:03:13.728872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.728836    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.728872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.728877    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.728892    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.729042    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.729077    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.729166    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.729219    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.729283    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.729332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.729304    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gb9\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.830521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830487    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.830730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830531    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.830730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830586    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.830730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830604    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gb9\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.830730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830645    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830719    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830791    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830876    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.830900    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.831081    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.831174    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.831234    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.831326    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.831465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.831444    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.833395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.833371    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.833508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.833491    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.883860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.883781    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.884039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.883987    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gb9\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9\") pod \"e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") " pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:13.913858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:13.913801    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:14.326311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.326285    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"]
Nov 03 12:03:14.328145 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:14.328116    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449e7deb_c747_49e0_a99b_be7a0041bcb3.slice/crio-eaf843dfc42707450049a562a96f3f4988cbe185ea5ff149814038c4b1c13c80 WatchSource:0}: Error finding container eaf843dfc42707450049a562a96f3f4988cbe185ea5ff149814038c4b1c13c80: Status 404 returned error can't find the container with id eaf843dfc42707450049a562a96f3f4988cbe185ea5ff149814038c4b1c13c80
Nov 03 12:03:14.330653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.330621    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:14.330781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.330696    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:14.330781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.330741    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:14.488477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.488442    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" event={"ID":"449e7deb-c747-49e0-a99b-be7a0041bcb3","Type":"ContainerStarted","Data":"ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"}
Nov 03 12:03:14.488643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.488505    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" event={"ID":"449e7deb-c747-49e0-a99b-be7a0041bcb3","Type":"ContainerStarted","Data":"eaf843dfc42707450049a562a96f3f4988cbe185ea5ff149814038c4b1c13c80"}
Nov 03 12:03:14.676565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.676442    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podStartSLOduration=1.67642516 podStartE2EDuration="1.67642516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:14.605154061 +0000 UTC m=+736.176738007" watchObservedRunningTime="2026-11-03 12:03:14.67642516 +0000 UTC m=+736.248009071"
Nov 03 12:03:14.678178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.678153    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"]
Nov 03 12:03:14.681691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.681668    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.730593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.730563    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-30e6ec98\""
Nov 03 12:03:14.730593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.730583    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-30e6ec98-dockercfg-djlbx\""
Nov 03 12:03:14.736114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.736087    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"]
Nov 03 12:03:14.842051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.842011    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kds\" (UniqueName: \"kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds\") pod \"payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.842183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.842072    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.914453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.914421    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:14.943711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.943623    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kds\" (UniqueName: \"kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds\") pod \"payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.943711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.943698    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.944382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.944362    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.997260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.997229    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kds\" (UniqueName: \"kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds\") pod \"payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:14.998041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:14.998014    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:15.364244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:15.364163    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"]
Nov 03 12:03:15.366564 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:15.366531    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f47e9e_9251_4ea0_b05c_5671b39c83a5.slice/crio-d651e40ed4825e68335bb1ae921f62506786cb0603ee849c63e75b0806b6f966 WatchSource:0}: Error finding container d651e40ed4825e68335bb1ae921f62506786cb0603ee849c63e75b0806b6f966: Status 404 returned error can't find the container with id d651e40ed4825e68335bb1ae921f62506786cb0603ee849c63e75b0806b6f966
Nov 03 12:03:15.494257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:15.494224    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2" event={"ID":"69f47e9e-9251-4ea0-b05c-5671b39c83a5","Type":"ContainerStarted","Data":"fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18"}
Nov 03 12:03:15.494425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:15.494278    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2" event={"ID":"69f47e9e-9251-4ea0-b05c-5671b39c83a5","Type":"ContainerStarted","Data":"d651e40ed4825e68335bb1ae921f62506786cb0603ee849c63e75b0806b6f966"}
Nov 03 12:03:15.603076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:15.603025    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2" podStartSLOduration=1.603011478 podStartE2EDuration="1.603011478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:15.601337945 +0000 UTC m=+737.172921856" watchObservedRunningTime="2026-11-03 12:03:15.603011478 +0000 UTC m=+737.174595389"
Nov 03 12:03:15.920057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:15.920019    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:16.500004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:16.499970    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:16.500219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:16.500033    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:16.500915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:16.500894    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:19.011707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.011674    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"]
Nov 03 12:03:19.015333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.015308    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.064440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.064408    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1319e7ef\""
Nov 03 12:03:19.086708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.086672    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.086878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.086741    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnrc\" (UniqueName: \"kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc\") pod \"payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.110901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.110873    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1319e7ef-dockercfg-7jzck\""
Nov 03 12:03:19.112481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.112463    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"]
Nov 03 12:03:19.187568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.187529    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.187720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.187591    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnrc\" (UniqueName: \"kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc\") pod \"payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.188325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.188301    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.240587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.240557    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnrc\" (UniqueName: \"kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc\") pod \"payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.328545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.328472    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:19.692411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:19.692384    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"]
Nov 03 12:03:19.695278 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:19.695253    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55745b0_9727_493d_ae30_ec4f8ed15cdd.slice/crio-0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d WatchSource:0}: Error finding container 0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d: Status 404 returned error can't find the container with id 0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d
Nov 03 12:03:20.519013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:20.518976    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln" event={"ID":"d55745b0-9727-493d-ae30-ec4f8ed15cdd","Type":"ContainerStarted","Data":"6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513"}
Nov 03 12:03:20.519390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:20.519036    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln" event={"ID":"d55745b0-9727-493d-ae30-ec4f8ed15cdd","Type":"ContainerStarted","Data":"0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d"}
Nov 03 12:03:20.519390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:20.519068    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:20.634559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:20.634512    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln" podStartSLOduration=2.634498675 podStartE2EDuration="2.634498675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:20.634455183 +0000 UTC m=+742.206039113" watchObservedRunningTime="2026-11-03 12:03:20.634498675 +0000 UTC m=+742.206082583"
Nov 03 12:03:21.279427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.279400    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"]
Nov 03 12:03:21.283060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.283041    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.309991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.309970    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6628\" (UniqueName: \"kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628\") pod \"payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.310108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.310086    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.331963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.331935    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f732077f-dockercfg-9rflh\""
Nov 03 12:03:21.332063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.331962    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f732077f\""
Nov 03 12:03:21.338641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.338612    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"]
Nov 03 12:03:21.410990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.410960    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.411159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.411031    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6628\" (UniqueName: \"kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628\") pod \"payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.411598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.411579    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.464627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.464596    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6628\" (UniqueName: \"kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628\") pod \"payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.596148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.596072    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:21.961008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:21.960986    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"]
Nov 03 12:03:21.963196 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:21.963170    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41f3ee4_a825_4594_a3d9_7911f9ac72d7.slice/crio-1df066e4dbc27d01f060d9c796932da54869234df5ca57f821a733af34335c4c WatchSource:0}: Error finding container 1df066e4dbc27d01f060d9c796932da54869234df5ca57f821a733af34335c4c: Status 404 returned error can't find the container with id 1df066e4dbc27d01f060d9c796932da54869234df5ca57f821a733af34335c4c
Nov 03 12:03:22.531612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:22.531580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26" event={"ID":"e41f3ee4-a825-4594-a3d9-7911f9ac72d7","Type":"ContainerStarted","Data":"4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893"}
Nov 03 12:03:22.531836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:22.531626    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26" event={"ID":"e41f3ee4-a825-4594-a3d9-7911f9ac72d7","Type":"ContainerStarted","Data":"1df066e4dbc27d01f060d9c796932da54869234df5ca57f821a733af34335c4c"}
Nov 03 12:03:22.531836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:22.531661    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:22.639190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:22.639145    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26" podStartSLOduration=1.6391296629999998 podStartE2EDuration="1.639129663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:22.63753919 +0000 UTC m=+744.209123127" watchObservedRunningTime="2026-11-03 12:03:22.639129663 +0000 UTC m=+744.210713576"
Nov 03 12:03:23.622535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.622501    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"]
Nov 03 12:03:23.626236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.626217    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.629670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.629647    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hbh\" (UniqueName: \"kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh\") pod \"payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.629804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.629720    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.725070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.725032    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4b9370d1\""
Nov 03 12:03:23.725547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.725147    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4b9370d1-dockercfg-7dj8f\""
Nov 03 12:03:23.729957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.728357    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"]
Nov 03 12:03:23.731078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.731047    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hbh\" (UniqueName: \"kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh\") pod \"payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.731436 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.731408    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.732413 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.732382    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.784738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.784710    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"]
Nov 03 12:03:23.787308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.787282    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hbh\" (UniqueName: \"kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh\") pod \"payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.788336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.788321    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:23.832549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.832527    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:23.832725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.832703    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crfsr\" (UniqueName: \"kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr\") pod \"payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:23.842318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.842293    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"]
Nov 03 12:03:23.933823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.933721    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crfsr\" (UniqueName: \"kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr\") pod \"payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:23.933823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.933818    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:23.934542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.934516    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:23.938604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.938585    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:23.988461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:23.988387    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crfsr\" (UniqueName: \"kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr\") pod \"payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:24.108178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.108138    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:24.301057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.301015    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"]
Nov 03 12:03:24.303304 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:24.303278    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd23b716_1eea_42d9_b4f6_442d0c3c4ad5.slice/crio-95a7aa78f1be754dc8f441c6cb7239df1a21af7755486922bbb587e3f72bacf3 WatchSource:0}: Error finding container 95a7aa78f1be754dc8f441c6cb7239df1a21af7755486922bbb587e3f72bacf3: Status 404 returned error can't find the container with id 95a7aa78f1be754dc8f441c6cb7239df1a21af7755486922bbb587e3f72bacf3
Nov 03 12:03:24.471318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.471298    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"]
Nov 03 12:03:24.473443 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:24.473412    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6f0a37_23b1_4f14_8846_19491fbb7cfd.slice/crio-de26177b9ff0d9a57ed8fceb342fea41573f3b7ace04a72d4864fb5ef0a25ee9 WatchSource:0}: Error finding container de26177b9ff0d9a57ed8fceb342fea41573f3b7ace04a72d4864fb5ef0a25ee9: Status 404 returned error can't find the container with id de26177b9ff0d9a57ed8fceb342fea41573f3b7ace04a72d4864fb5ef0a25ee9
Nov 03 12:03:24.549810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.549736    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz" event={"ID":"2d6f0a37-23b1-4f14-8846-19491fbb7cfd","Type":"ContainerStarted","Data":"0ebd633535a9df9a697e49159caa5532fcf8d7f72631a535bcd4a627897195ae"}
Nov 03 12:03:24.549991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.549875    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz" event={"ID":"2d6f0a37-23b1-4f14-8846-19491fbb7cfd","Type":"ContainerStarted","Data":"de26177b9ff0d9a57ed8fceb342fea41573f3b7ace04a72d4864fb5ef0a25ee9"}
Nov 03 12:03:24.549991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.549908    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:24.551474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.551397    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx" event={"ID":"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5","Type":"ContainerStarted","Data":"4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0"}
Nov 03 12:03:24.551474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.551450    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:24.551662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.551474    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx" event={"ID":"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5","Type":"ContainerStarted","Data":"95a7aa78f1be754dc8f441c6cb7239df1a21af7755486922bbb587e3f72bacf3"}
Nov 03 12:03:24.660915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.660869    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz" podStartSLOduration=1.6608556650000001 podStartE2EDuration="1.660855665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:24.658136014 +0000 UTC m=+746.229719942" watchObservedRunningTime="2026-11-03 12:03:24.660855665 +0000 UTC m=+746.232439598"
Nov 03 12:03:24.764955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:24.764898    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx" podStartSLOduration=1.764882726 podStartE2EDuration="1.764882726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:24.763692693 +0000 UTC m=+746.335276621" watchObservedRunningTime="2026-11-03 12:03:24.764882726 +0000 UTC m=+746.336466637"
Nov 03 12:03:27.531994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:27.531958    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:03:31.532759 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:31.532729    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:03:32.320934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.320891    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"]
Nov 03 12:03:32.324795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.324773    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.376392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.376353    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9ab28c7a\""
Nov 03 12:03:32.422533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.422493    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9ab28c7a-dockercfg-qrrd2\""
Nov 03 12:03:32.425972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.425929    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"]
Nov 03 12:03:32.429205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.429009    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.429205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.429058    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c556\" (UniqueName: \"kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556\") pod \"payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.530546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.530501    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.530737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.530555    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c556\" (UniqueName: \"kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556\") pod \"payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.531215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.531196    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.536554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.536522    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"]
Nov 03 12:03:32.540305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.540283    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.589378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.589284    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c556\" (UniqueName: \"kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556\") pod \"payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.608047 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.608015    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"]
Nov 03 12:03:32.632056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.632019    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume\") pod \"payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") " pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.632246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.632107    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hch8b\" (UniqueName: \"kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b\") pod \"payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") " pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.639822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.639791    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:32.733903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.733862    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume\") pod \"payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") " pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.734053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.733921    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hch8b\" (UniqueName: \"kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b\") pod \"payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") " pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.735263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.735230    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume\") pod \"payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") " pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.789007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.788969    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hch8b\" (UniqueName: \"kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b\") pod \"payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") " pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:32.853321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:32.853230    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:33.027156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.027134    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"]
Nov 03 12:03:33.030256 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:33.030215    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4c5b7_e248_4adb_bb28_68c4eabf3be5.slice/crio-2f916ebab9c13d67b61869636f804f90a90af00a7cecb7d69c0431a0c972b0fb WatchSource:0}: Error finding container 2f916ebab9c13d67b61869636f804f90a90af00a7cecb7d69c0431a0c972b0fb: Status 404 returned error can't find the container with id 2f916ebab9c13d67b61869636f804f90a90af00a7cecb7d69c0431a0c972b0fb
Nov 03 12:03:33.231279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.230228    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"]
Nov 03 12:03:33.239395 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:33.239001    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dca8f3_588a_42a9_82f2_eda3abab53fd.slice/crio-a148ed73a8952d1f533170a6faec75798f04329d11c9d4b4959ad930964690a8 WatchSource:0}: Error finding container a148ed73a8952d1f533170a6faec75798f04329d11c9d4b4959ad930964690a8: Status 404 returned error can't find the container with id a148ed73a8952d1f533170a6faec75798f04329d11c9d4b4959ad930964690a8
Nov 03 12:03:33.540260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.540169    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:03:33.597551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.597515    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x" event={"ID":"c8dca8f3-588a-42a9-82f2-eda3abab53fd","Type":"ContainerStarted","Data":"3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9"}
Nov 03 12:03:33.597770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.597569    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x" event={"ID":"c8dca8f3-588a-42a9-82f2-eda3abab53fd","Type":"ContainerStarted","Data":"a148ed73a8952d1f533170a6faec75798f04329d11c9d4b4959ad930964690a8"}
Nov 03 12:03:33.597770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.597611    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:33.599370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.599339    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd" event={"ID":"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5","Type":"ContainerStarted","Data":"fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23"}
Nov 03 12:03:33.599488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.599390    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd" event={"ID":"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5","Type":"ContainerStarted","Data":"2f916ebab9c13d67b61869636f804f90a90af00a7cecb7d69c0431a0c972b0fb"}
Nov 03 12:03:33.599488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.599427    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:33.756178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.756100    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd" podStartSLOduration=1.756080577 podStartE2EDuration="1.756080577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:33.754705415 +0000 UTC m=+755.326289329" watchObservedRunningTime="2026-11-03 12:03:33.756080577 +0000 UTC m=+755.327664491"
Nov 03 12:03:33.861145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:33.861032    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x" podStartSLOduration=1.8610141150000001 podStartE2EDuration="1.861014115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:33.859809249 +0000 UTC m=+755.431393172" watchObservedRunningTime="2026-11-03 12:03:33.861014115 +0000 UTC m=+755.432598031"
Nov 03 12:03:35.559021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.558993    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:03:35.559568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.559056    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:03:35.634511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.634480    4546 generic.go:356] "Generic (PLEG): container finished" podID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerID="3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6" exitCode=0
Nov 03 12:03:35.634670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.634565    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" event={"ID":"e163e1eb-3878-4636-8feb-9d89f1d7ce8d","Type":"ContainerDied","Data":"3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"}
Nov 03 12:03:35.635038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.635021    4546 scope.go:122] "RemoveContainer" containerID="3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"
Nov 03 12:03:35.636327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.636300    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:35.636431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.636383    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:35.636493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:35.636431    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:36.657946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:36.657902    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" event={"ID":"e163e1eb-3878-4636-8feb-9d89f1d7ce8d","Type":"ContainerStarted","Data":"bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db"}
Nov 03 12:03:37.229774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:37.229379    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:37.231925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:37.231891    4546 patch_prober.go:29] interesting pod/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 12:03:37.232197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:37.232167    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 12:03:37.677424 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:03:37.677320    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:03:38.227774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:38.227229    4546 patch_prober.go:29] interesting pod/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 12:03:38.227774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:38.227308    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 12:03:39.235774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:39.227206    4546 patch_prober.go:29] interesting pod/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 12:03:39.235774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:39.227302    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 12:03:39.460470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:39.460437    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:40.228775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:40.225172    4546 patch_prober.go:29] interesting pod/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 12:03:40.228775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:40.225241    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 12:03:41.229776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:41.228783    4546 patch_prober.go:29] interesting pod/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 12:03:41.229776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:41.228849    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 12:03:42.226035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.226004    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:42.227148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.227122    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:03:42.515296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.508107    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:03:42.515296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.508186    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:03:42.690017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.689934    4546 generic.go:356] "Generic (PLEG): container finished" podID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerID="47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9" exitCode=0
Nov 03 12:03:42.690160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.690010    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" event={"ID":"c6997879-6745-4d61-b4f8-12f5e9420d58","Type":"ContainerDied","Data":"47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"}
Nov 03 12:03:42.690512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.690492    4546 scope.go:122] "RemoveContainer" containerID="47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"
Nov 03 12:03:42.691691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.691662    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:42.691793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.691741    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:42.691841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:42.691824    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:43.696999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:43.696967    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" event={"ID":"c6997879-6745-4d61-b4f8-12f5e9420d58","Type":"ContainerStarted","Data":"a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114"}
Nov 03 12:03:44.195944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:44.195913    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:44.196160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:44.196139    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:03:44.196213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:44.196185    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 12:03:44.632393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:44.632366    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:03:44.632596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:44.632440    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:03:45.195561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:45.195511    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:03:45.196018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:45.195591    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 12:03:46.196141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:46.196103    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:03:46.196604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:46.196575    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 12:03:47.195875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:47.195839    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:03:47.196067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:47.195913    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 12:03:48.195403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:48.195369    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 12:03:48.195835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:48.195421    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 12:03:49.226665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:49.226593    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:49.227232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:49.227185    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:49.228139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:49.228119    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:03:52.001660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.001626    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"]
Nov 03 12:03:52.005723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.005701    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.054570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.054543    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:03:52.100385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.100351    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:03:52.100522 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.100395    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:03:52.100522 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.100458    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:03:52.102429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.102411    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"]
Nov 03 12:03:52.170693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.170661    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdv22\" (UniqueName: \"kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.170693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.170696    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.170919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.170873    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.271442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.271371    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.271584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.271456    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdv22\" (UniqueName: \"kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.271584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.271478    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.271984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.271964    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.273868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.273840    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.347441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.347409    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdv22\" (UniqueName: \"kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22\") pod \"maas-api-revoke-keys-e2e-ait-f732077f-kpfz8\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:52.642010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:52.641929    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:53.004886 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:53.004863    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"]
Nov 03 12:03:53.006698 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:03:53.006670    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf81473_c80d_48a7_81e4_8a3e9d56a373.slice/crio-0991692173eb5f7e570361f4330bae164a327d608bb06fc38b8a254b9f88813b WatchSource:0}: Error finding container 0991692173eb5f7e570361f4330bae164a327d608bb06fc38b8a254b9f88813b: Status 404 returned error can't find the container with id 0991692173eb5f7e570361f4330bae164a327d608bb06fc38b8a254b9f88813b
Nov 03 12:03:53.771345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:53.771305    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8" event={"ID":"caf81473-c80d-48a7-81e4-8a3e9d56a373","Type":"ContainerStarted","Data":"11dcdc3956bcbdeefa619c739ef9f08d8c0674a7418f5f07f9b871236f551987"}
Nov 03 12:03:53.771537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:53.771367    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8" event={"ID":"caf81473-c80d-48a7-81e4-8a3e9d56a373","Type":"ContainerStarted","Data":"0991692173eb5f7e570361f4330bae164a327d608bb06fc38b8a254b9f88813b"}
Nov 03 12:03:53.882277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:53.882225    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8" podStartSLOduration=2.88221044 podStartE2EDuration="2.88221044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:53.88132945 +0000 UTC m=+775.452913361" watchObservedRunningTime="2026-11-03 12:03:53.88221044 +0000 UTC m=+775.453794355"
Nov 03 12:03:54.777500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:54.777466    4546 generic.go:356] "Generic (PLEG): container finished" podID="caf81473-c80d-48a7-81e4-8a3e9d56a373" containerID="11dcdc3956bcbdeefa619c739ef9f08d8c0674a7418f5f07f9b871236f551987" exitCode=0
Nov 03 12:03:54.777925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:54.777540    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8" event={"ID":"caf81473-c80d-48a7-81e4-8a3e9d56a373","Type":"ContainerDied","Data":"11dcdc3956bcbdeefa619c739ef9f08d8c0674a7418f5f07f9b871236f551987"}
Nov 03 12:03:55.923476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:55.923445    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:56.012090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.012058    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca\") pod \"caf81473-c80d-48a7-81e4-8a3e9d56a373\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") "
Nov 03 12:03:56.012247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.012138    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22\" (UniqueName: \"kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22\") pod \"caf81473-c80d-48a7-81e4-8a3e9d56a373\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") "
Nov 03 12:03:56.012247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.012169    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca\") pod \"caf81473-c80d-48a7-81e4-8a3e9d56a373\" (UID: \"caf81473-c80d-48a7-81e4-8a3e9d56a373\") "
Nov 03 12:03:56.012645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.012606    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca" pod "caf81473-c80d-48a7-81e4-8a3e9d56a373" (UID: "caf81473-c80d-48a7-81e4-8a3e9d56a373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:03:56.014265 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.014236    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca" pod "caf81473-c80d-48a7-81e4-8a3e9d56a373" (UID: "caf81473-c80d-48a7-81e4-8a3e9d56a373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:03:56.014420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.014398    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22" pod "caf81473-c80d-48a7-81e4-8a3e9d56a373" (UID: "caf81473-c80d-48a7-81e4-8a3e9d56a373"). InnerVolumeSpecName "kube-api-access-wdv22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:03:56.113954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.113865    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdv22\" (UniqueName: \"kubernetes.io/projected/caf81473-c80d-48a7-81e4-8a3e9d56a373-kube-api-access-wdv22\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:03:56.113954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.113907    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:03:56.113954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.113924    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf81473-c80d-48a7-81e4-8a3e9d56a373-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:03:56.791194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.791156    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8" event={"ID":"caf81473-c80d-48a7-81e4-8a3e9d56a373","Type":"ContainerDied","Data":"0991692173eb5f7e570361f4330bae164a327d608bb06fc38b8a254b9f88813b"}
Nov 03 12:03:56.791194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.791195    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"
Nov 03 12:03:56.791194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:56.791203    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0991692173eb5f7e570361f4330bae164a327d608bb06fc38b8a254b9f88813b"
Nov 03 12:03:58.810134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:58.810100    4546 generic.go:356] "Generic (PLEG): container finished" podID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerID="ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2" exitCode=0
Nov 03 12:03:58.810563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:58.810174    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" event={"ID":"449e7deb-c747-49e0-a99b-be7a0041bcb3","Type":"ContainerDied","Data":"ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"}
Nov 03 12:03:58.810642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:58.810626    4546 scope.go:122] "RemoveContainer" containerID="ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"
Nov 03 12:03:58.811933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:58.811908    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:58.812013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:58.811972    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:58.812013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:58.812001    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:03:59.356431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.356392    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"]
Nov 03 12:03:59.356877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.356863    4546 cpu_manager.go:414] "containerMap: removing container" podUID="caf81473-c80d-48a7-81e4-8a3e9d56a373" containerName="revoke-keys"
Nov 03 12:03:59.356930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.356878    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf81473-c80d-48a7-81e4-8a3e9d56a373" containerName="revoke-keys"
Nov 03 12:03:59.356962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.356955    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf81473-c80d-48a7-81e4-8a3e9d56a373" containerName="revoke-keys"
Nov 03 12:03:59.360238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.360217    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.423773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.417527    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"]
Nov 03 12:03:59.423773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.419915    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:03:59.423773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.420038    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:03:59.423773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.420160    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:03:59.423773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.420206    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:03:59.455418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.455377    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zvm\" (UniqueName: \"kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.455565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.455522    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.455565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.455549    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.556433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.556387    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.556433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.556437    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.556695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.556508    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zvm\" (UniqueName: \"kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.557133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.557108    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.559066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.559041    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.618494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.618371    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zvm\" (UniqueName: \"kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm\") pod \"maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.672609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.672575    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:03:59.837438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.837410    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" event={"ID":"449e7deb-c747-49e0-a99b-be7a0041bcb3","Type":"ContainerStarted","Data":"e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61"}
Nov 03 12:03:59.932074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.931975    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:03:59.933358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.933321    4546 patch_prober.go:29] interesting pod/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:03:59.933541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:03:59.933372    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 12:04:00.086660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:00.086629    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"]
Nov 03 12:04:00.088521 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:00.088482    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec08370_a4d2_4635_9a2a_23825f7b57e8.slice/crio-26fcaafb594d23e5edc6e9da45118c91d98a9ab7f8b0ec120fc2cee6a155bf49 WatchSource:0}: Error finding container 26fcaafb594d23e5edc6e9da45118c91d98a9ab7f8b0ec120fc2cee6a155bf49: Status 404 returned error can't find the container with id 26fcaafb594d23e5edc6e9da45118c91d98a9ab7f8b0ec120fc2cee6a155bf49
Nov 03 12:04:00.844035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:00.844000    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b" event={"ID":"6ec08370-a4d2-4635-9a2a-23825f7b57e8","Type":"ContainerStarted","Data":"c809ee5a0f7b1cf4be2e17723d96f0e1e5a444ea02e44d200c0bf3dbd0ce4afd"}
Nov 03 12:04:00.844445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:00.844056    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b" event={"ID":"6ec08370-a4d2-4635-9a2a-23825f7b57e8","Type":"ContainerStarted","Data":"26fcaafb594d23e5edc6e9da45118c91d98a9ab7f8b0ec120fc2cee6a155bf49"}
Nov 03 12:04:00.925776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:00.924202    4546 patch_prober.go:29] interesting pod/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:04:00.925776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:00.924270    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 12:04:00.952495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:00.952440    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b" podStartSLOduration=1.952423249 podStartE2EDuration="1.952423249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:00.950217468 +0000 UTC m=+782.521801380" watchObservedRunningTime="2026-11-03 12:04:00.952423249 +0000 UTC m=+782.524007187"
Nov 03 12:04:01.500430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:01.500397    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:04:01.851343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:01.851257    4546 generic.go:356] "Generic (PLEG): container finished" podID="6ec08370-a4d2-4635-9a2a-23825f7b57e8" containerID="c809ee5a0f7b1cf4be2e17723d96f0e1e5a444ea02e44d200c0bf3dbd0ce4afd" exitCode=0
Nov 03 12:04:01.851343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:01.851301    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b" event={"ID":"6ec08370-a4d2-4635-9a2a-23825f7b57e8","Type":"ContainerDied","Data":"c809ee5a0f7b1cf4be2e17723d96f0e1e5a444ea02e44d200c0bf3dbd0ce4afd"}
Nov 03 12:04:01.932310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:01.932267    4546 patch_prober.go:29] interesting pod/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:04:01.932700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:01.932335    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 12:04:02.928171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:02.927708    4546 patch_prober.go:29] interesting pod/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:04:02.928171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:02.927790    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 12:04:03.039550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.039520    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:04:03.192919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.192839    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca\") pod \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") "
Nov 03 12:04:03.192919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.192886    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm\" (UniqueName: \"kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm\") pod \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") "
Nov 03 12:04:03.192919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.192906    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca\") pod \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\" (UID: \"6ec08370-a4d2-4635-9a2a-23825f7b57e8\") "
Nov 03 12:04:03.193273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.193249    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca" pod "6ec08370-a4d2-4635-9a2a-23825f7b57e8" (UID: "6ec08370-a4d2-4635-9a2a-23825f7b57e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:03.195173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.195149    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm" pod "6ec08370-a4d2-4635-9a2a-23825f7b57e8" (UID: "6ec08370-a4d2-4635-9a2a-23825f7b57e8"). InnerVolumeSpecName "kube-api-access-62zvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:03.195324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.195304    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca" pod "6ec08370-a4d2-4635-9a2a-23825f7b57e8" (UID: "6ec08370-a4d2-4635-9a2a-23825f7b57e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:03.294590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.294554    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62zvm\" (UniqueName: \"kubernetes.io/projected/6ec08370-a4d2-4635-9a2a-23825f7b57e8-kube-api-access-62zvm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:03.294590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.294586    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:03.294590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.294597    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec08370-a4d2-4635-9a2a-23825f7b57e8-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:03.346887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.346853    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"]
Nov 03 12:04:03.863950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.863922    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"
Nov 03 12:04:03.864151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.863959    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b" event={"ID":"6ec08370-a4d2-4635-9a2a-23825f7b57e8","Type":"ContainerDied","Data":"26fcaafb594d23e5edc6e9da45118c91d98a9ab7f8b0ec120fc2cee6a155bf49"}
Nov 03 12:04:03.864151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.864005    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26fcaafb594d23e5edc6e9da45118c91d98a9ab7f8b0ec120fc2cee6a155bf49"
Nov 03 12:04:03.929307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.929271    4546 patch_prober.go:29] interesting pod/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 12:04:03.930433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:03.930005    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 12:04:04.989838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:04.989800    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:04:04.991695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:04.991673    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:04:07.651544 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:07.651504    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:04:08.347661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.347613    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26" podUID="e41f3ee4-a825-4594-a3d9-7911f9ac72d7" containerName="payload-pre-processing" containerID="cri-o://4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893" gracePeriod=25
Nov 03 12:04:08.609468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.609397    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:04:08.760769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.760719    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628\" (UniqueName: \"kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628\") pod \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") "
Nov 03 12:04:08.761171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.760812    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume\") pod \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\" (UID: \"e41f3ee4-a825-4594-a3d9-7911f9ac72d7\") "
Nov 03 12:04:08.761171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.761143    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume" pod "e41f3ee4-a825-4594-a3d9-7911f9ac72d7" (UID: "e41f3ee4-a825-4594-a3d9-7911f9ac72d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:08.763230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.763200    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628" pod "e41f3ee4-a825-4594-a3d9-7911f9ac72d7" (UID: "e41f3ee4-a825-4594-a3d9-7911f9ac72d7"). InnerVolumeSpecName "kube-api-access-p6628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:08.861608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.861524    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6628\" (UniqueName: \"kubernetes.io/projected/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-kube-api-access-p6628\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:08.861608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.861551    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f3ee4-a825-4594-a3d9-7911f9ac72d7-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:08.889075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.889046    4546 generic.go:356] "Generic (PLEG): container finished" podID="e41f3ee4-a825-4594-a3d9-7911f9ac72d7" containerID="4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893" exitCode=0
Nov 03 12:04:08.889267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.889093    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"
Nov 03 12:04:08.889267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.889110    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26" event={"ID":"e41f3ee4-a825-4594-a3d9-7911f9ac72d7","Type":"ContainerDied","Data":"4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893"}
Nov 03 12:04:08.889267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.889151    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26" event={"ID":"e41f3ee4-a825-4594-a3d9-7911f9ac72d7","Type":"ContainerDied","Data":"1df066e4dbc27d01f060d9c796932da54869234df5ca57f821a733af34335c4c"}
Nov 03 12:04:08.889267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.889173    4546 scope.go:122] "RemoveContainer" containerID="4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893"
Nov 03 12:04:08.898646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.898627    4546 scope.go:122] "RemoveContainer" containerID="4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893"
Nov 03 12:04:08.899007 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:08.898985    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893\": container with ID starting with 4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893 not found: ID does not exist" containerID="4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893"
Nov 03 12:04:08.899057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:08.899019    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893"} err="failed to get container status \"4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893\": rpc error: code = NotFound desc = could not find container \"4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893\": container with ID starting with 4e877787245df3fe9368e5c09f422abd1dc701f96678afff47cb87cc19a18893 not found: ID does not exist"
Nov 03 12:04:09.044531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:09.044495    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"]
Nov 03 12:04:09.048162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:09.048139    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f732077f-7966f855-gmg26"]
Nov 03 12:04:10.571671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:10.571627    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41f3ee4-a825-4594-a3d9-7911f9ac72d7" path="/var/lib/kubelet/pods/e41f3ee4-a825-4594-a3d9-7911f9ac72d7/volumes"
Nov 03 12:04:10.777643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:10.777601    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"]
Nov 03 12:04:10.876986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:10.876903    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"]
Nov 03 12:04:12.609705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.609670    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"]
Nov 03 12:04:12.610134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.610098    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e41f3ee4-a825-4594-a3d9-7911f9ac72d7" containerName="payload-pre-processing"
Nov 03 12:04:12.610134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.610110    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41f3ee4-a825-4594-a3d9-7911f9ac72d7" containerName="payload-pre-processing"
Nov 03 12:04:12.610134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.610120    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6ec08370-a4d2-4635-9a2a-23825f7b57e8" containerName="revoke-keys"
Nov 03 12:04:12.610134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.610125    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec08370-a4d2-4635-9a2a-23825f7b57e8" containerName="revoke-keys"
Nov 03 12:04:12.610261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.610214    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41f3ee4-a825-4594-a3d9-7911f9ac72d7" containerName="payload-pre-processing"
Nov 03 12:04:12.610261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.610222    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec08370-a4d2-4635-9a2a-23825f7b57e8" containerName="revoke-keys"
Nov 03 12:04:12.613476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.613458    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.697016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.696981    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.697198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.697093    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.697198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.697150    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbf76\" (UniqueName: \"kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.711276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.711241    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:04:12.711276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.711275    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:04:12.711530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.711284    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:04:12.711530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.711325    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:04:12.713468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.713441    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"]
Nov 03 12:04:12.798182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.798140    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.798403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.798222    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.798403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.798270    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbf76\" (UniqueName: \"kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.798943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.798918    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.800817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.800795    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.852027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.851988    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbf76\" (UniqueName: \"kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76\") pod \"maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:12.926591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:12.926503    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:13.311708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.311674    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"]
Nov 03 12:04:13.320147 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:13.320108    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08721e7_1855_4bf5_8b3b_c200e3d61eee.slice/crio-1f5cacbc92e4462c74a82fe869cbf35d5c4ab232720d5022073ed56cf51b51ab WatchSource:0}: Error finding container 1f5cacbc92e4462c74a82fe869cbf35d5c4ab232720d5022073ed56cf51b51ab: Status 404 returned error can't find the container with id 1f5cacbc92e4462c74a82fe869cbf35d5c4ab232720d5022073ed56cf51b51ab
Nov 03 12:04:13.596088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.595991    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"]
Nov 03 12:04:13.598318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.598289    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f732077f-kpfz8"]
Nov 03 12:04:13.646531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.646495    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"]
Nov 03 12:04:13.647025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.646801    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy" containerID="cri-o://bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db" gracePeriod=30
Nov 03 12:04:13.916657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.916572    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4" event={"ID":"f08721e7-1855-4bf5-8b3b-c200e3d61eee","Type":"ContainerStarted","Data":"7af54fa2c812603c4dd0112f7b5e3e2c2a9c959e8b625bca9a007160608b1fd7"}
Nov 03 12:04:13.916657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:13.916621    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4" event={"ID":"f08721e7-1855-4bf5-8b3b-c200e3d61eee","Type":"ContainerStarted","Data":"1f5cacbc92e4462c74a82fe869cbf35d5c4ab232720d5022073ed56cf51b51ab"}
Nov 03 12:04:14.023046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:14.022998    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4" podStartSLOduration=2.022986009 podStartE2EDuration="2.022986009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:14.022237503 +0000 UTC m=+795.593821416" watchObservedRunningTime="2026-11-03 12:04:14.022986009 +0000 UTC m=+795.594569992"
Nov 03 12:04:14.571919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:14.571886    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf81473-c80d-48a7-81e4-8a3e9d56a373" path="/var/lib/kubelet/pods/caf81473-c80d-48a7-81e4-8a3e9d56a373/volumes"
Nov 03 12:04:14.923302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:14.923202    4546 generic.go:356] "Generic (PLEG): container finished" podID="f08721e7-1855-4bf5-8b3b-c200e3d61eee" containerID="7af54fa2c812603c4dd0112f7b5e3e2c2a9c959e8b625bca9a007160608b1fd7" exitCode=0
Nov 03 12:04:14.923689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:14.923307    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4" event={"ID":"f08721e7-1855-4bf5-8b3b-c200e3d61eee","Type":"ContainerDied","Data":"7af54fa2c812603c4dd0112f7b5e3e2c2a9c959e8b625bca9a007160608b1fd7"}
Nov 03 12:04:15.778625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:15.778549    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz" podUID="2d6f0a37-23b1-4f14-8846-19491fbb7cfd" containerName="payload-processing" containerID="cri-o://0ebd633535a9df9a697e49159caa5532fcf8d7f72631a535bcd4a627897195ae" gracePeriod=25
Nov 03 12:04:15.877415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:15.877374    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx" podUID="bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" containerName="payload-pre-processing" containerID="cri-o://4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0" gracePeriod=25
Nov 03 12:04:15.930440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:15.930402    4546 generic.go:356] "Generic (PLEG): container finished" podID="2d6f0a37-23b1-4f14-8846-19491fbb7cfd" containerID="0ebd633535a9df9a697e49159caa5532fcf8d7f72631a535bcd4a627897195ae" exitCode=0
Nov 03 12:04:15.930841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:15.930445    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz" event={"ID":"2d6f0a37-23b1-4f14-8846-19491fbb7cfd","Type":"ContainerDied","Data":"0ebd633535a9df9a697e49159caa5532fcf8d7f72631a535bcd4a627897195ae"}
Nov 03 12:04:16.046253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.046225    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:04:16.106367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.106336    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:16.141001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.140976    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume\") pod \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") "
Nov 03 12:04:16.141153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141073    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr\" (UniqueName: \"kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr\") pod \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\" (UID: \"2d6f0a37-23b1-4f14-8846-19491fbb7cfd\") "
Nov 03 12:04:16.141221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141154    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca\") pod \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") "
Nov 03 12:04:16.141221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141204    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76\" (UniqueName: \"kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76\") pod \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") "
Nov 03 12:04:16.141331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141259    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca\") pod \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\" (UID: \"f08721e7-1855-4bf5-8b3b-c200e3d61eee\") "
Nov 03 12:04:16.141377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141323    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume" pod "2d6f0a37-23b1-4f14-8846-19491fbb7cfd" (UID: "2d6f0a37-23b1-4f14-8846-19491fbb7cfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:16.141556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141540    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.141707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.141680    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca" pod "f08721e7-1855-4bf5-8b3b-c200e3d61eee" (UID: "f08721e7-1855-4bf5-8b3b-c200e3d61eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:16.143527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.143492    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca" pod "f08721e7-1855-4bf5-8b3b-c200e3d61eee" (UID: "f08721e7-1855-4bf5-8b3b-c200e3d61eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:16.143527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.143496    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76" pod "f08721e7-1855-4bf5-8b3b-c200e3d61eee" (UID: "f08721e7-1855-4bf5-8b3b-c200e3d61eee"). InnerVolumeSpecName "kube-api-access-nbf76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:16.143719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.143546    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr" pod "2d6f0a37-23b1-4f14-8846-19491fbb7cfd" (UID: "2d6f0a37-23b1-4f14-8846-19491fbb7cfd"). InnerVolumeSpecName "kube-api-access-crfsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:16.146402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.146378    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:04:16.241919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.241885    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume\") pod \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") "
Nov 03 12:04:16.242097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.241930    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh\" (UniqueName: \"kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh\") pod \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\" (UID: \"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5\") "
Nov 03 12:04:16.242207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.242196    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.242250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.242210    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbf76\" (UniqueName: \"kubernetes.io/projected/f08721e7-1855-4bf5-8b3b-c200e3d61eee-kube-api-access-nbf76\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.242250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.242220    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08721e7-1855-4bf5-8b3b-c200e3d61eee-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.242250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.242229    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crfsr\" (UniqueName: \"kubernetes.io/projected/2d6f0a37-23b1-4f14-8846-19491fbb7cfd-kube-api-access-crfsr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.242250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.242236    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume" pod "bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" (UID: "bd23b716-1eea-42d9-b4f6-442d0c3c4ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:16.244126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.244104    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh" pod "bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" (UID: "bd23b716-1eea-42d9-b4f6-442d0c3c4ad5"). InnerVolumeSpecName "kube-api-access-z9hbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:16.343023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.342948    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.343023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.342973    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hbh\" (UniqueName: \"kubernetes.io/projected/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5-kube-api-access-z9hbh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:16.355393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.355366    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"]
Nov 03 12:04:16.936971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.936940    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"
Nov 03 12:04:16.936971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.936955    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz" event={"ID":"2d6f0a37-23b1-4f14-8846-19491fbb7cfd","Type":"ContainerDied","Data":"de26177b9ff0d9a57ed8fceb342fea41573f3b7ace04a72d4864fb5ef0a25ee9"}
Nov 03 12:04:16.937459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.937011    4546 scope.go:122] "RemoveContainer" containerID="0ebd633535a9df9a697e49159caa5532fcf8d7f72631a535bcd4a627897195ae"
Nov 03 12:04:16.938727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.938709    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"
Nov 03 12:04:16.938859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.938710    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4" event={"ID":"f08721e7-1855-4bf5-8b3b-c200e3d61eee","Type":"ContainerDied","Data":"1f5cacbc92e4462c74a82fe869cbf35d5c4ab232720d5022073ed56cf51b51ab"}
Nov 03 12:04:16.938859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.938821    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f5cacbc92e4462c74a82fe869cbf35d5c4ab232720d5022073ed56cf51b51ab"
Nov 03 12:04:16.940348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.940330    4546 generic.go:356] "Generic (PLEG): container finished" podID="bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" containerID="4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0" exitCode=0
Nov 03 12:04:16.940518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.940376    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"
Nov 03 12:04:16.940518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.940393    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx" event={"ID":"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5","Type":"ContainerDied","Data":"4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0"}
Nov 03 12:04:16.940518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.940432    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx" event={"ID":"bd23b716-1eea-42d9-b4f6-442d0c3c4ad5","Type":"ContainerDied","Data":"95a7aa78f1be754dc8f441c6cb7239df1a21af7755486922bbb587e3f72bacf3"}
Nov 03 12:04:16.947430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.947410    4546 scope.go:122] "RemoveContainer" containerID="4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0"
Nov 03 12:04:16.955631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.955615    4546 scope.go:122] "RemoveContainer" containerID="4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0"
Nov 03 12:04:16.955967 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:16.955948    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0\": container with ID starting with 4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0 not found: ID does not exist" containerID="4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0"
Nov 03 12:04:16.956024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:16.955974    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0"} err="failed to get container status \"4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0\": rpc error: code = NotFound desc = could not find container \"4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0\": container with ID starting with 4b8456fb425787fae7f417ec20e3408f5d8805a42fc003b3b8564322899b00a0 not found: ID does not exist"
Nov 03 12:04:17.143130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.143035    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"]
Nov 03 12:04:17.143130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.143094    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b9370d1-5d6ccbdbd7-dp5dx"]
Nov 03 12:04:17.241443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.241415    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"]
Nov 03 12:04:17.247565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.247538    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b9370d1-6f86448b9f-xklwz"]
Nov 03 12:04:17.672981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.672946    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"]
Nov 03 12:04:17.673398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673385    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" containerName="payload-pre-processing"
Nov 03 12:04:17.673456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673401    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" containerName="payload-pre-processing"
Nov 03 12:04:17.673456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673422    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2d6f0a37-23b1-4f14-8846-19491fbb7cfd" containerName="payload-processing"
Nov 03 12:04:17.673456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673428    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6f0a37-23b1-4f14-8846-19491fbb7cfd" containerName="payload-processing"
Nov 03 12:04:17.673549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673461    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f08721e7-1855-4bf5-8b3b-c200e3d61eee" containerName="revoke-keys"
Nov 03 12:04:17.673549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673466    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08721e7-1855-4bf5-8b3b-c200e3d61eee" containerName="revoke-keys"
Nov 03 12:04:17.673549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673530    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6f0a37-23b1-4f14-8846-19491fbb7cfd" containerName="payload-processing"
Nov 03 12:04:17.673549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673544    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08721e7-1855-4bf5-8b3b-c200e3d61eee" containerName="revoke-keys"
Nov 03 12:04:17.673549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.673550    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" containerName="payload-pre-processing"
Nov 03 12:04:17.678299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.678276    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.728533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.728501    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:04:17.728737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.728568    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:04:17.728737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.728575    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:04:17.728737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.728679    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:04:17.731715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.731688    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"]
Nov 03 12:04:17.758807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.758736    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.758994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.758879    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvx5x\" (UniqueName: \"kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.758994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.758967    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.860296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.860258    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.860453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.860318    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvx5x\" (UniqueName: \"kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.860453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.860391    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.861186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.861106    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.862884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.862863    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.914392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.914360    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvx5x\" (UniqueName: \"kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x\") pod \"maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:17.993324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:17.993294    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:18.360254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.360233    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"]
Nov 03 12:04:18.362229 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:18.362186    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c66391_ee70_4c99_9a15_02e2b18eb0b1.slice/crio-43e491f3f3239d9f631ea7e96b889715d423de5e44cd6d440a02852c70e48881 WatchSource:0}: Error finding container 43e491f3f3239d9f631ea7e96b889715d423de5e44cd6d440a02852c70e48881: Status 404 returned error can't find the container with id 43e491f3f3239d9f631ea7e96b889715d423de5e44cd6d440a02852c70e48881
Nov 03 12:04:18.572175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.572144    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6f0a37-23b1-4f14-8846-19491fbb7cfd" path="/var/lib/kubelet/pods/2d6f0a37-23b1-4f14-8846-19491fbb7cfd/volumes"
Nov 03 12:04:18.572515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.572503    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd23b716-1eea-42d9-b4f6-442d0c3c4ad5" path="/var/lib/kubelet/pods/bd23b716-1eea-42d9-b4f6-442d0c3c4ad5/volumes"
Nov 03 12:04:18.913458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.913434    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:04:18.953647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.953612    4546 generic.go:356] "Generic (PLEG): container finished" podID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerID="bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db" exitCode=0
Nov 03 12:04:18.953838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.953658    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"
Nov 03 12:04:18.953838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.953701    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" event={"ID":"e163e1eb-3878-4636-8feb-9d89f1d7ce8d","Type":"ContainerDied","Data":"bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db"}
Nov 03 12:04:18.953838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.953781    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz" event={"ID":"e163e1eb-3878-4636-8feb-9d89f1d7ce8d","Type":"ContainerDied","Data":"143a86a5582c19295756d14a42f01498e97f719f64319e02cc419257b27081f4"}
Nov 03 12:04:18.953838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.953814    4546 scope.go:122] "RemoveContainer" containerID="bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db"
Nov 03 12:04:18.955304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.955283    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg" event={"ID":"72c66391-ee70-4c99-9a15-02e2b18eb0b1","Type":"ContainerStarted","Data":"8c61fc924d38d0c21f0a484ae4b4e491e9cf0a72c014d8746fc07987e40c092a"}
Nov 03 12:04:18.955422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.955325    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg" event={"ID":"72c66391-ee70-4c99-9a15-02e2b18eb0b1","Type":"ContainerStarted","Data":"43e491f3f3239d9f631ea7e96b889715d423de5e44cd6d440a02852c70e48881"}
Nov 03 12:04:18.962535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.962518    4546 scope.go:122] "RemoveContainer" containerID="3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"
Nov 03 12:04:18.970961 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.970943    4546 scope.go:122] "RemoveContainer" containerID="bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db"
Nov 03 12:04:18.971216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971201    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971251 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:18.971198    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db\": container with ID starting with bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db not found: ID does not exist" containerID="bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db"
Nov 03 12:04:18.971295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971253    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db"} err="failed to get container status \"bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db\": rpc error: code = NotFound desc = could not find container \"bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db\": container with ID starting with bb63c9a5e3a9bfbc23f0e9eb3addb2de0671f1fefff5b145b5d52a166e4d01db not found: ID does not exist"
Nov 03 12:04:18.971295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971273    4546 scope.go:122] "RemoveContainer" containerID="3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"
Nov 03 12:04:18.971371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971289    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971327    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971358    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971395    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971433    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971469    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971541    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971630    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket\") pod \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\" (UID: \"e163e1eb-3878-4636-8feb-9d89f1d7ce8d\") "
Nov 03 12:04:18.971782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971646    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:18.971782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971727    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:18.971782 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:18.971736    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6\": container with ID starting with 3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6 not found: ID does not exist" containerID="3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"
Nov 03 12:04:18.971938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.971801    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6"} err="failed to get container status \"3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6\": rpc error: code = NotFound desc = could not find container \"3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6\": container with ID starting with 3972921f332958ef1c2c5ebd47cc3a0fa8b4b0df3d57284c00bb7408c2695ba6 not found: ID does not exist"
Nov 03 12:04:18.972221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.972018    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:18.972221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.972042    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:18.972221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.972049    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:18.972221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.972189    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:18.972221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.972214    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:18.973466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.973438    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:04:18.973566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.973544    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:18.974014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.973989    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "kube-api-access-zb67d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:18.974309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:18.974293    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token" pod "e163e1eb-3878-4636-8feb-9d89f1d7ce8d" (UID: "e163e1eb-3878-4636-8feb-9d89f1d7ce8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:19.072714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072677    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb67d\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-kube-api-access-zb67d\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.072714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072703    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.072714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072714    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.072714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072723    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.073329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072732    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.073329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072740    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.073329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.072776    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e163e1eb-3878-4636-8feb-9d89f1d7ce8d-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:19.177502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.177389    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg" podStartSLOduration=2.177374761 podStartE2EDuration="2.177374761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:19.174699007 +0000 UTC m=+800.746282932" watchObservedRunningTime="2026-11-03 12:04:19.177374761 +0000 UTC m=+800.748958672"
Nov 03 12:04:19.368515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.368480    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"]
Nov 03 12:04:19.373594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.373559    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f732077f-openshift-default-68c5fb4b4f-t96tz"]
Nov 03 12:04:19.962404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.962371    4546 generic.go:356] "Generic (PLEG): container finished" podID="72c66391-ee70-4c99-9a15-02e2b18eb0b1" containerID="8c61fc924d38d0c21f0a484ae4b4e491e9cf0a72c014d8746fc07987e40c092a" exitCode=0
Nov 03 12:04:19.962564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:19.962427    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg" event={"ID":"72c66391-ee70-4c99-9a15-02e2b18eb0b1","Type":"ContainerDied","Data":"8c61fc924d38d0c21f0a484ae4b4e491e9cf0a72c014d8746fc07987e40c092a"}
Nov 03 12:04:20.571300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:20.571265    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" path="/var/lib/kubelet/pods/e163e1eb-3878-4636-8feb-9d89f1d7ce8d/volumes"
Nov 03 12:04:21.095424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.095400    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:21.194873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.194845    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca\") pod \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") "
Nov 03 12:04:21.195026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.194902    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca\") pod \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") "
Nov 03 12:04:21.195026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.194932    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x\" (UniqueName: \"kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x\") pod \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\" (UID: \"72c66391-ee70-4c99-9a15-02e2b18eb0b1\") "
Nov 03 12:04:21.195180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.195123    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca" pod "72c66391-ee70-4c99-9a15-02e2b18eb0b1" (UID: "72c66391-ee70-4c99-9a15-02e2b18eb0b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:21.197178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.197153    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x" pod "72c66391-ee70-4c99-9a15-02e2b18eb0b1" (UID: "72c66391-ee70-4c99-9a15-02e2b18eb0b1"). InnerVolumeSpecName "kube-api-access-hvx5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:21.197178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.197167    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca" pod "72c66391-ee70-4c99-9a15-02e2b18eb0b1" (UID: "72c66391-ee70-4c99-9a15-02e2b18eb0b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:21.297283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.297191    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:21.297283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.297230    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c66391-ee70-4c99-9a15-02e2b18eb0b1-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:21.297283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.297246    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvx5x\" (UniqueName: \"kubernetes.io/projected/72c66391-ee70-4c99-9a15-02e2b18eb0b1-kube-api-access-hvx5x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:21.356469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.356405    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2" podUID="69f47e9e-9251-4ea0-b05c-5671b39c83a5" containerName="payload-processing" containerID="cri-o://fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18" gracePeriod=25
Nov 03 12:04:21.591237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.591212    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:04:21.700972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.700932    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume\") pod \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") "
Nov 03 12:04:21.701209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.700990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds\" (UniqueName: \"kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds\") pod \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\" (UID: \"69f47e9e-9251-4ea0-b05c-5671b39c83a5\") "
Nov 03 12:04:21.701607 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.701582    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume" pod "69f47e9e-9251-4ea0-b05c-5671b39c83a5" (UID: "69f47e9e-9251-4ea0-b05c-5671b39c83a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:21.703726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.703695    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds" pod "69f47e9e-9251-4ea0-b05c-5671b39c83a5" (UID: "69f47e9e-9251-4ea0-b05c-5671b39c83a5"). InnerVolumeSpecName "kube-api-access-g2kds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:21.802349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.802312    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f47e9e-9251-4ea0-b05c-5671b39c83a5-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:21.802349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.802347    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kds\" (UniqueName: \"kubernetes.io/projected/69f47e9e-9251-4ea0-b05c-5671b39c83a5-kube-api-access-g2kds\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:21.973987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.973951    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"
Nov 03 12:04:21.973987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.973980    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg" event={"ID":"72c66391-ee70-4c99-9a15-02e2b18eb0b1","Type":"ContainerDied","Data":"43e491f3f3239d9f631ea7e96b889715d423de5e44cd6d440a02852c70e48881"}
Nov 03 12:04:21.974225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.974033    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e491f3f3239d9f631ea7e96b889715d423de5e44cd6d440a02852c70e48881"
Nov 03 12:04:21.975460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.975428    4546 generic.go:356] "Generic (PLEG): container finished" podID="69f47e9e-9251-4ea0-b05c-5671b39c83a5" containerID="fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18" exitCode=0
Nov 03 12:04:21.975584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.975472    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"
Nov 03 12:04:21.975584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.975510    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2" event={"ID":"69f47e9e-9251-4ea0-b05c-5671b39c83a5","Type":"ContainerDied","Data":"fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18"}
Nov 03 12:04:21.975584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.975561    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2" event={"ID":"69f47e9e-9251-4ea0-b05c-5671b39c83a5","Type":"ContainerDied","Data":"d651e40ed4825e68335bb1ae921f62506786cb0603ee849c63e75b0806b6f966"}
Nov 03 12:04:21.975712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.975588    4546 scope.go:122] "RemoveContainer" containerID="fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18"
Nov 03 12:04:21.986091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.986059    4546 scope.go:122] "RemoveContainer" containerID="fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18"
Nov 03 12:04:21.986383 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:21.986356    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18\": container with ID starting with fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18 not found: ID does not exist" containerID="fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18"
Nov 03 12:04:21.986431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:21.986394    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18"} err="failed to get container status \"fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18\": rpc error: code = NotFound desc = could not find container \"fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18\": container with ID starting with fe07a02c86905ab3f46bc7f9f71fb710c3319ec492fd85c6eb8d28a60b5b0b18 not found: ID does not exist"
Nov 03 12:04:22.175072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:22.175034    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"]
Nov 03 12:04:22.178893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:22.178869    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-30e6ec98-5847495dcf-8qjf2"]
Nov 03 12:04:22.571963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:22.571932    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f47e9e-9251-4ea0-b05c-5671b39c83a5" path="/var/lib/kubelet/pods/69f47e9e-9251-4ea0-b05c-5671b39c83a5/volumes"
Nov 03 12:04:23.084308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084276    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"]
Nov 03 12:04:23.084696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084662    4546 cpu_manager.go:414] "containerMap: removing container" podUID="69f47e9e-9251-4ea0-b05c-5671b39c83a5" containerName="payload-processing"
Nov 03 12:04:23.084696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084672    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f47e9e-9251-4ea0-b05c-5671b39c83a5" containerName="payload-processing"
Nov 03 12:04:23.084696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084692    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy"
Nov 03 12:04:23.084696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084697    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084709    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084714    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084720    4546 cpu_manager.go:414] "containerMap: removing container" podUID="72c66391-ee70-4c99-9a15-02e2b18eb0b1" containerName="revoke-keys"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084725    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c66391-ee70-4c99-9a15-02e2b18eb0b1" containerName="revoke-keys"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084795    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084803    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c66391-ee70-4c99-9a15-02e2b18eb0b1" containerName="revoke-keys"
Nov 03 12:04:23.084876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.084811    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f47e9e-9251-4ea0-b05c-5671b39c83a5" containerName="payload-processing"
Nov 03 12:04:23.089202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.089175    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.140836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.140790    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-91cd99\""
Nov 03 12:04:23.140836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.140790    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-91cd99-dockercfg-6qjh9\""
Nov 03 12:04:23.177763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.177708    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"]
Nov 03 12:04:23.217210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.217174    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.217366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.217340    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfcb\" (UniqueName: \"kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb\") pod \"payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.246862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.246826    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"]
Nov 03 12:04:23.247309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.247297    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e163e1eb-3878-4636-8feb-9d89f1d7ce8d" containerName="istio-proxy"
Nov 03 12:04:23.250376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.250357    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.306793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.306735    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"]
Nov 03 12:04:23.329349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.329307    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.329559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.329413    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfcb\" (UniqueName: \"kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb\") pod \"payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.329559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.329473    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4hq\" (UniqueName: \"kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq\") pod \"payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.329559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.329510    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.330456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.330423    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.383553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.383471    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfcb\" (UniqueName: \"kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb\") pod \"payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.402286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.402258    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:23.430758 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.430717    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4hq\" (UniqueName: \"kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq\") pod \"payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.430907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.430822    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.431471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.431451    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.484186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.484158    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4hq\" (UniqueName: \"kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq\") pod \"payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.562537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.562501    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:23.580956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.580919    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"]
Nov 03 12:04:23.794073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.793869    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"]
Nov 03 12:04:23.796647 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:23.796615    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cd0866_d324_4d00_8dd8_89c5590d815b.slice/crio-1afc776f38db083d917c9a7343db593beea17c7894a66ab5e595bc3ef7bd2d46 WatchSource:0}: Error finding container 1afc776f38db083d917c9a7343db593beea17c7894a66ab5e595bc3ef7bd2d46: Status 404 returned error can't find the container with id 1afc776f38db083d917c9a7343db593beea17c7894a66ab5e595bc3ef7bd2d46
Nov 03 12:04:23.932660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.932599    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"]
Nov 03 12:04:23.934886 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:23.934854    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376faac6_7b43_410b_bde9_1544dc445a93.slice/crio-f4ba15c1fb1460c78318db712b8e25c0573c34f54736649e5352321d898f1da9 WatchSource:0}: Error finding container f4ba15c1fb1460c78318db712b8e25c0573c34f54736649e5352321d898f1da9: Status 404 returned error can't find the container with id f4ba15c1fb1460c78318db712b8e25c0573c34f54736649e5352321d898f1da9
Nov 03 12:04:23.990956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.990924    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" event={"ID":"376faac6-7b43-410b-bde9-1544dc445a93","Type":"ContainerStarted","Data":"f4ba15c1fb1460c78318db712b8e25c0573c34f54736649e5352321d898f1da9"}
Nov 03 12:04:23.992880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.992853    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" event={"ID":"86cd0866-d324-4d00-8dd8-89c5590d815b","Type":"ContainerStarted","Data":"30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271"}
Nov 03 12:04:23.993016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.992906    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" event={"ID":"86cd0866-d324-4d00-8dd8-89c5590d815b","Type":"ContainerStarted","Data":"1afc776f38db083d917c9a7343db593beea17c7894a66ab5e595bc3ef7bd2d46"}
Nov 03 12:04:23.993016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:23.992973    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:24.101874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:24.101742    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" podStartSLOduration=1.101727191 podStartE2EDuration="1.101727191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:24.099707706 +0000 UTC m=+805.671291638" watchObservedRunningTime="2026-11-03 12:04:24.101727191 +0000 UTC m=+805.673311103"
Nov 03 12:04:25.000224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:25.000184    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" event={"ID":"376faac6-7b43-410b-bde9-1544dc445a93","Type":"ContainerStarted","Data":"1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0"}
Nov 03 12:04:25.000547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:25.000526    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:25.106876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:25.106830    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" podStartSLOduration=2.106815265 podStartE2EDuration="2.106815265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:25.105144079 +0000 UTC m=+806.676728010" watchObservedRunningTime="2026-11-03 12:04:25.106815265 +0000 UTC m=+806.678399204"
Nov 03 12:04:28.582202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:28.582162    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln" podUID="d55745b0-9727-493d-ae30-ec4f8ed15cdd" containerName="payload-pre-processing" containerID="cri-o://6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513" gracePeriod=25
Nov 03 12:04:28.837652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:28.837592    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:04:28.985399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:28.985367    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc\" (UniqueName: \"kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc\") pod \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") "
Nov 03 12:04:28.985572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:28.985459    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume\") pod \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\" (UID: \"d55745b0-9727-493d-ae30-ec4f8ed15cdd\") "
Nov 03 12:04:28.985832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:28.985809    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume" pod "d55745b0-9727-493d-ae30-ec4f8ed15cdd" (UID: "d55745b0-9727-493d-ae30-ec4f8ed15cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:28.987627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:28.987603    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc" pod "d55745b0-9727-493d-ae30-ec4f8ed15cdd" (UID: "d55745b0-9727-493d-ae30-ec4f8ed15cdd"). InnerVolumeSpecName "kube-api-access-hwnrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:29.022141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.022116    4546 generic.go:356] "Generic (PLEG): container finished" podID="d55745b0-9727-493d-ae30-ec4f8ed15cdd" containerID="6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513" exitCode=0
Nov 03 12:04:29.022267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.022159    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"
Nov 03 12:04:29.022267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.022203    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln" event={"ID":"d55745b0-9727-493d-ae30-ec4f8ed15cdd","Type":"ContainerDied","Data":"6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513"}
Nov 03 12:04:29.022267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.022261    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln" event={"ID":"d55745b0-9727-493d-ae30-ec4f8ed15cdd","Type":"ContainerDied","Data":"0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d"}
Nov 03 12:04:29.022391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.022289    4546 scope.go:122] "RemoveContainer" containerID="6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513"
Nov 03 12:04:29.032606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.032589    4546 scope.go:122] "RemoveContainer" containerID="6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513"
Nov 03 12:04:29.032890 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:29.032871    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513\": container with ID starting with 6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513 not found: ID does not exist" containerID="6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513"
Nov 03 12:04:29.032968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.032899    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513"} err="failed to get container status \"6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513\": rpc error: code = NotFound desc = could not find container \"6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513\": container with ID starting with 6fb139606f07394a530e5751dc9c41cf467579e3a095bc69b6009dbad4cad513 not found: ID does not exist"
Nov 03 12:04:29.054720 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:29.054688    4546 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55745b0_9727_493d_ae30_ec4f8ed15cdd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55745b0_9727_493d_ae30_ec4f8ed15cdd.slice/crio-0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d\": RecentStats: unable to find data in memory cache]"
Nov 03 12:04:29.054838 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:29.054799    4546 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55745b0_9727_493d_ae30_ec4f8ed15cdd.slice/crio-0c2b53153c4beb8d07cf1308d89a90df49ec04f5b39bff1f1940e12d1a02178d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55745b0_9727_493d_ae30_ec4f8ed15cdd.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 12:04:29.086244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.086219    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwnrc\" (UniqueName: \"kubernetes.io/projected/d55745b0-9727-493d-ae30-ec4f8ed15cdd-kube-api-access-hwnrc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:29.086339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.086248    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55745b0-9727-493d-ae30-ec4f8ed15cdd-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:29.174524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.174455    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"]
Nov 03 12:04:29.180081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.180060    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1319e7ef-bff6bb799-6qhln"]
Nov 03 12:04:29.212637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.212616    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"]
Nov 03 12:04:29.305890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:29.305869    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"]
Nov 03 12:04:30.071577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:30.071543    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"]
Nov 03 12:04:30.073950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:30.073930    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1319e7ef-9fhr4"]
Nov 03 12:04:30.574765 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:30.574727    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55745b0-9727-493d-ae30-ec4f8ed15cdd" path="/var/lib/kubelet/pods/d55745b0-9727-493d-ae30-ec4f8ed15cdd/volumes"
Nov 03 12:04:30.575216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:30.575195    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08721e7-1855-4bf5-8b3b-c200e3d61eee" path="/var/lib/kubelet/pods/f08721e7-1855-4bf5-8b3b-c200e3d61eee/volumes"
Nov 03 12:04:31.064702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:31.064673    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"]
Nov 03 12:04:31.110372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:31.110343    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b9370d1-r5t5b"]
Nov 03 12:04:31.646640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:31.646602    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"]
Nov 03 12:04:31.646882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:31.646856    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" containerID="cri-o://a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114" gracePeriod=30
Nov 03 12:04:32.572295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:32.572262    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec08370-a4d2-4635-9a2a-23825f7b57e8" path="/var/lib/kubelet/pods/6ec08370-a4d2-4635-9a2a-23825f7b57e8/volumes"
Nov 03 12:04:33.013685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.013656    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"]
Nov 03 12:04:33.014105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.014092    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d55745b0-9727-493d-ae30-ec4f8ed15cdd" containerName="payload-pre-processing"
Nov 03 12:04:33.014151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.014107    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55745b0-9727-493d-ae30-ec4f8ed15cdd" containerName="payload-pre-processing"
Nov 03 12:04:33.014185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.014166    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55745b0-9727-493d-ae30-ec4f8ed15cdd" containerName="payload-pre-processing"
Nov 03 12:04:33.018492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.018471    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.071779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.071711    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d2c7c-openshift-default-dockercfg-tjwjb\""
Nov 03 12:04:33.115794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.115735    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"]
Nov 03 12:04:33.127033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127001    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127275    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127331    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8d85\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127416    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127509    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127576    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127686    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127709    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.127776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.127728    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229304    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229366    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229438    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229528    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229552    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229581    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229619    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229653    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.229830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.229701    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8d85\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.230080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.230054    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.230403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.230373    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.230616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.230596    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.230792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.230771    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.231342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.231317    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.233089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.233062    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.234063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.234035    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.291648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.291573    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8d85\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.294271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.294249    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.339804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.339773    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:33.765667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.765611    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"]
Nov 03 12:04:33.768166 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:33.768134    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b43c591_0ba5_419f_8bf0_55c164c532e2.slice/crio-14bdc6cddd09f1fe015c61e7fcd1dc4c13f7eea2c0abf7fabe93bfcb800a227e WatchSource:0}: Error finding container 14bdc6cddd09f1fe015c61e7fcd1dc4c13f7eea2c0abf7fabe93bfcb800a227e: Status 404 returned error can't find the container with id 14bdc6cddd09f1fe015c61e7fcd1dc4c13f7eea2c0abf7fabe93bfcb800a227e
Nov 03 12:04:33.770739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.770705    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:04:33.770881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.770801    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:04:33.770881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:33.770850    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:04:34.060299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.060181    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" event={"ID":"9b43c591-0ba5-419f-8bf0-55c164c532e2","Type":"ContainerStarted","Data":"37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"}
Nov 03 12:04:34.060299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.060244    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" event={"ID":"9b43c591-0ba5-419f-8bf0-55c164c532e2","Type":"ContainerStarted","Data":"14bdc6cddd09f1fe015c61e7fcd1dc4c13f7eea2c0abf7fabe93bfcb800a227e"}
Nov 03 12:04:34.170397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.170336    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podStartSLOduration=2.170315063 podStartE2EDuration="2.170315063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:34.167852713 +0000 UTC m=+815.739436645" watchObservedRunningTime="2026-11-03 12:04:34.170315063 +0000 UTC m=+815.741898976"
Nov 03 12:04:34.213856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.213773    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x" podUID="c8dca8f3-588a-42a9-82f2-eda3abab53fd" containerName="payload-processing" containerID="cri-o://3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9" gracePeriod=25
Nov 03 12:04:34.229643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.229612    4546 patch_prober.go:29] interesting pod/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:04:34.229775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.229666    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:04:34.307071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.307006    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd" podUID="4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" containerName="payload-pre-processing" containerID="cri-o://fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23" gracePeriod=25
Nov 03 12:04:34.340759 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.340710    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:34.346296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.346271    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:34.567735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.567659    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:04:34.591385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.591362    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:04:34.745109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.745079    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume\") pod \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") "
Nov 03 12:04:34.745109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.745113    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b\" (UniqueName: \"kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b\") pod \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") "
Nov 03 12:04:34.745356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.745141    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume\") pod \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\" (UID: \"c8dca8f3-588a-42a9-82f2-eda3abab53fd\") "
Nov 03 12:04:34.745356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.745208    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556\" (UniqueName: \"kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556\") pod \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\" (UID: \"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5\") "
Nov 03 12:04:34.745610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.745573    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume" pod "4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" (UID: "4ba4c5b7-e248-4adb-bb28-68c4eabf3be5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:34.745610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.745600    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume" pod "c8dca8f3-588a-42a9-82f2-eda3abab53fd" (UID: "c8dca8f3-588a-42a9-82f2-eda3abab53fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:34.747544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.747520    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b" pod "c8dca8f3-588a-42a9-82f2-eda3abab53fd" (UID: "c8dca8f3-588a-42a9-82f2-eda3abab53fd"). InnerVolumeSpecName "kube-api-access-hch8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:34.747627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.747541    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556" pod "4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" (UID: "4ba4c5b7-e248-4adb-bb28-68c4eabf3be5"). InnerVolumeSpecName "kube-api-access-8c556". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:34.846638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.846539    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:34.846638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.846576    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hch8b\" (UniqueName: \"kubernetes.io/projected/c8dca8f3-588a-42a9-82f2-eda3abab53fd-kube-api-access-hch8b\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:34.846638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.846590    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8dca8f3-588a-42a9-82f2-eda3abab53fd-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:34.846638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:34.846602    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c556\" (UniqueName: \"kubernetes.io/projected/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5-kube-api-access-8c556\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:35.002879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.002849    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 12:04:35.066027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.065998    4546 generic.go:356] "Generic (PLEG): container finished" podID="c8dca8f3-588a-42a9-82f2-eda3abab53fd" containerID="3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9" exitCode=0
Nov 03 12:04:35.066237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.066044    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"
Nov 03 12:04:35.066237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.066079    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x" event={"ID":"c8dca8f3-588a-42a9-82f2-eda3abab53fd","Type":"ContainerDied","Data":"3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9"}
Nov 03 12:04:35.066237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.066132    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x" event={"ID":"c8dca8f3-588a-42a9-82f2-eda3abab53fd","Type":"ContainerDied","Data":"a148ed73a8952d1f533170a6faec75798f04329d11c9d4b4959ad930964690a8"}
Nov 03 12:04:35.066237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.066164    4546 scope.go:122] "RemoveContainer" containerID="3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9"
Nov 03 12:04:35.067486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.067462    4546 generic.go:356] "Generic (PLEG): container finished" podID="4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" containerID="fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23" exitCode=0
Nov 03 12:04:35.067573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.067507    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"
Nov 03 12:04:35.067573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.067535    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd" event={"ID":"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5","Type":"ContainerDied","Data":"fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23"}
Nov 03 12:04:35.067656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.067576    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd" event={"ID":"4ba4c5b7-e248-4adb-bb28-68c4eabf3be5","Type":"ContainerDied","Data":"2f916ebab9c13d67b61869636f804f90a90af00a7cecb7d69c0431a0c972b0fb"}
Nov 03 12:04:35.068005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.067954    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:35.073820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.073798    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:04:35.076769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.076705    4546 scope.go:122] "RemoveContainer" containerID="3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9"
Nov 03 12:04:35.077159 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:35.077137    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9\": container with ID starting with 3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9 not found: ID does not exist" containerID="3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9"
Nov 03 12:04:35.077302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.077166    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9"} err="failed to get container status \"3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9\": rpc error: code = NotFound desc = could not find container \"3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9\": container with ID starting with 3edd8961b5885b6f31e5ca85c3ef5f5e1e5d8642bf48bb7b00165b6ca037ddb9 not found: ID does not exist"
Nov 03 12:04:35.077302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.077189    4546 scope.go:122] "RemoveContainer" containerID="fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23"
Nov 03 12:04:35.087506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.087480    4546 scope.go:122] "RemoveContainer" containerID="fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23"
Nov 03 12:04:35.088073 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:35.088045    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23\": container with ID starting with fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23 not found: ID does not exist" containerID="fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23"
Nov 03 12:04:35.088166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.088085    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23"} err="failed to get container status \"fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23\": rpc error: code = NotFound desc = could not find container \"fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23\": container with ID starting with fdebbc5025aa971bedc50d53b4bc39b78c5a9ab3558127ca98ea989ceb923f23 not found: ID does not exist"
Nov 03 12:04:35.216584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.216552    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"]
Nov 03 12:04:35.225811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.225780    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9ab28c7a-5cccd44c77-m8h6x"]
Nov 03 12:04:35.233218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233182    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"]
Nov 03 12:04:35.233597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233582    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" containerName="payload-pre-processing"
Nov 03 12:04:35.233638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233599    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" containerName="payload-pre-processing"
Nov 03 12:04:35.233638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233615    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c8dca8f3-588a-42a9-82f2-eda3abab53fd" containerName="payload-processing"
Nov 03 12:04:35.233638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233623    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8dca8f3-588a-42a9-82f2-eda3abab53fd" containerName="payload-processing"
Nov 03 12:04:35.233730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233697    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" containerName="payload-pre-processing"
Nov 03 12:04:35.233730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.233706    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8dca8f3-588a-42a9-82f2-eda3abab53fd" containerName="payload-processing"
Nov 03 12:04:35.238121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.238105    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.286669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.286643    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bb1b51-openshift-default-dockercfg-b9g7k\""
Nov 03 12:04:35.295109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.295067    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"]
Nov 03 12:04:35.351579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.351548    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncc78\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.351777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.351601    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.351777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.351682    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.351928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.351802    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.351928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.351852    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.351928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.351906    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.352037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.352004    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.352081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.352052    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.352113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.352089    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.437488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.437451    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"]
Nov 03 12:04:35.439236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.439210    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9ab28c7a-6dd8857575-qt4qd"]
Nov 03 12:04:35.452587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452559    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.452776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452599    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.452776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452732    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.452914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452809    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.452914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452849    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.452914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452872    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncc78\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452916    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452953    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.452969    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.453031    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.453109    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.453164    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.453186    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.453652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.453633    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.455104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.455081    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.455636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.455617    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.506707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.506631    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncc78\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.506862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.506839    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.551348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.551301    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:35.962518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.962487    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"]
Nov 03 12:04:35.963959 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:35.963917    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511d427a_a19d_411f_9c7c_2fd4674f2c73.slice/crio-771c75bbba3be0190ac957a2cf2d57d69158d3ad5fc79061ec6536c43bfd7992 WatchSource:0}: Error finding container 771c75bbba3be0190ac957a2cf2d57d69158d3ad5fc79061ec6536c43bfd7992: Status 404 returned error can't find the container with id 771c75bbba3be0190ac957a2cf2d57d69158d3ad5fc79061ec6536c43bfd7992
Nov 03 12:04:35.966446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.966415    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:04:35.966527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.966481    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:04:35.966527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:35.966511    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:04:36.009210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.009187    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 12:04:36.074295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.074257    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" event={"ID":"511d427a-a19d-411f-9c7c-2fd4674f2c73","Type":"ContainerStarted","Data":"f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"}
Nov 03 12:04:36.074468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.074328    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" event={"ID":"511d427a-a19d-411f-9c7c-2fd4674f2c73","Type":"ContainerStarted","Data":"771c75bbba3be0190ac957a2cf2d57d69158d3ad5fc79061ec6536c43bfd7992"}
Nov 03 12:04:36.228150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.228086    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podStartSLOduration=1.228063555 podStartE2EDuration="1.228063555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:36.227571258 +0000 UTC m=+817.799155175" watchObservedRunningTime="2026-11-03 12:04:36.228063555 +0000 UTC m=+817.799647468"
Nov 03 12:04:36.552022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.551943    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:36.572020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.571984    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba4c5b7-e248-4adb-bb28-68c4eabf3be5" path="/var/lib/kubelet/pods/4ba4c5b7-e248-4adb-bb28-68c4eabf3be5/volumes"
Nov 03 12:04:36.572403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.572389    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8dca8f3-588a-42a9-82f2-eda3abab53fd" path="/var/lib/kubelet/pods/c8dca8f3-588a-42a9-82f2-eda3abab53fd/volumes"
Nov 03 12:04:36.893637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:36.893614    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:04:37.069784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069691    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069803    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069832    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069862    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069900    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069951    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069989    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.069997    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070022    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070106    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.070210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070123    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo\") pod \"c6997879-6745-4d61-b4f8-12f5e9420d58\" (UID: \"c6997879-6745-4d61-b4f8-12f5e9420d58\") "
Nov 03 12:04:37.070738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070304    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.070738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070471    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:37.070738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070568    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.070738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070679    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.070738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070697    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6997879-6745-4d61-b4f8-12f5e9420d58-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.070738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070713    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.070990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.070740    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.072263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.072241    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:37.072343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.072283    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:37.072805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.072785    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "kube-api-access-w8dgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:37.072805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.072791    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo" pod "c6997879-6745-4d61-b4f8-12f5e9420d58" (UID: "c6997879-6745-4d61-b4f8-12f5e9420d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:04:37.083535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.083511    4546 generic.go:356] "Generic (PLEG): container finished" podID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerID="a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114" exitCode=0
Nov 03 12:04:37.083649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.083558    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"
Nov 03 12:04:37.083649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.083590    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" event={"ID":"c6997879-6745-4d61-b4f8-12f5e9420d58","Type":"ContainerDied","Data":"a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114"}
Nov 03 12:04:37.083782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.083655    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g" event={"ID":"c6997879-6745-4d61-b4f8-12f5e9420d58","Type":"ContainerDied","Data":"1650af3bcd76c0b0de46f3cc89dfa0e05d643ba139fa10bfda531108b9902b7d"}
Nov 03 12:04:37.083782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.083686    4546 scope.go:122] "RemoveContainer" containerID="a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114"
Nov 03 12:04:37.093902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.093877    4546 scope.go:122] "RemoveContainer" containerID="47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"
Nov 03 12:04:37.102291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.102271    4546 scope.go:122] "RemoveContainer" containerID="a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114"
Nov 03 12:04:37.102578 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:37.102559    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114\": container with ID starting with a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114 not found: ID does not exist" containerID="a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114"
Nov 03 12:04:37.102649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.102591    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114"} err="failed to get container status \"a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114\": rpc error: code = NotFound desc = could not find container \"a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114\": container with ID starting with a785b8b9c464fc8412996c1b8b16d9681beda3e7f9de2ecacc32e5c844cad114 not found: ID does not exist"
Nov 03 12:04:37.102649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.102609    4546 scope.go:122] "RemoveContainer" containerID="47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"
Nov 03 12:04:37.102873 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:37.102856    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9\": container with ID starting with 47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9 not found: ID does not exist" containerID="47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"
Nov 03 12:04:37.102926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.102880    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9"} err="failed to get container status \"47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9\": rpc error: code = NotFound desc = could not find container \"47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9\": container with ID starting with 47840ba3584deda6b0981144a3a6a275974765f889d754a83b1ac828eb6c42a9 not found: ID does not exist"
Nov 03 12:04:37.172094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.172062    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dgc\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-kube-api-access-w8dgc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.172094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.172092    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.172094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.172102    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.172346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.172112    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6997879-6745-4d61-b4f8-12f5e9420d58-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.172346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.172120    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6997879-6745-4d61-b4f8-12f5e9420d58-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:37.240886 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.240858    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"]
Nov 03 12:04:37.245800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.245773    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4b9370d1-openshift-default-b5bb4fc6f-8k78g"]
Nov 03 12:04:37.556835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:37.556799    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:37.649843 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:37.649807    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:04:38.090820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:38.090791    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:38.092124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:38.092095    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:04:38.573059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:38.573029    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" path="/var/lib/kubelet/pods/c6997879-6745-4d61-b4f8-12f5e9420d58/volumes"
Nov 03 12:04:43.601105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:43.601067    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"]
Nov 03 12:04:43.648531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:43.648484    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ab28c7a-8nlfg"]
Nov 03 12:04:43.670890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:43.670863    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"]
Nov 03 12:04:43.671132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:43.671105    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy" containerID="cri-o://e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61" gracePeriod=30
Nov 03 12:04:44.572769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:44.572711    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c66391-ee70-4c99-9a15-02e2b18eb0b1" path="/var/lib/kubelet/pods/72c66391-ee70-4c99-9a15-02e2b18eb0b1/volumes"
Nov 03 12:04:45.037492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.037452    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"]
Nov 03 12:04:45.038078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.038059    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy"
Nov 03 12:04:45.038184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.038080    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy"
Nov 03 12:04:45.038184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.038112    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy"
Nov 03 12:04:45.038184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.038120    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy"
Nov 03 12:04:45.038324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.038234    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy"
Nov 03 12:04:45.041597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.041576    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.089950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.089910    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bb1b51\""
Nov 03 12:04:45.119241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.119213    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bb1b51-dockercfg-pwvbk\""
Nov 03 12:04:45.137900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.137867    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"]
Nov 03 12:04:45.155850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.155812    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x4cl\" (UniqueName: \"kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl\") pod \"payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.156030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.155885    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.196988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.196932    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"]
Nov 03 12:04:45.198322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.197737    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6997879-6745-4d61-b4f8-12f5e9420d58" containerName="istio-proxy"
Nov 03 12:04:45.201499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.201469    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.253875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.253843    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"]
Nov 03 12:04:45.257353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.257324    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.257529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.257399    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.257666 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.257636    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x4cl\" (UniqueName: \"kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl\") pod \"payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.257798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.257679    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blrz\" (UniqueName: \"kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz\") pod \"payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.258094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.258074    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.315706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.315627    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x4cl\" (UniqueName: \"kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl\") pod \"payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.355558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.355522    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:45.359094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.358862    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blrz\" (UniqueName: \"kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz\") pod \"payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.359094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.358909    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.359849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.359825    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.412604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.412576    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blrz\" (UniqueName: \"kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz\") pod \"payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.522631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.522598    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:45.723472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.723428    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"]
Nov 03 12:04:45.725842 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:45.725808    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33009df_f592_4370_b7e0_caea97754eed.slice/crio-7f946d378759da4a8556c33e0e2edcf7c83274a346d277dfad406563ef58e130 WatchSource:0}: Error finding container 7f946d378759da4a8556c33e0e2edcf7c83274a346d277dfad406563ef58e130: Status 404 returned error can't find the container with id 7f946d378759da4a8556c33e0e2edcf7c83274a346d277dfad406563ef58e130
Nov 03 12:04:45.887354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:45.887178    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"]
Nov 03 12:04:45.890162 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:45.890126    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1eef8f_ba1b_4a10_a354_0087daf8d35a.slice/crio-db2b896f4ab9388dcae28c7814c138785a6cd60c6e746a501b1716102a83ced8 WatchSource:0}: Error finding container db2b896f4ab9388dcae28c7814c138785a6cd60c6e746a501b1716102a83ced8: Status 404 returned error can't find the container with id db2b896f4ab9388dcae28c7814c138785a6cd60c6e746a501b1716102a83ced8
Nov 03 12:04:46.136598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.136565    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" event={"ID":"2e1eef8f-ba1b-4a10-a354-0087daf8d35a","Type":"ContainerStarted","Data":"a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5"}
Nov 03 12:04:46.137078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.136617    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" event={"ID":"2e1eef8f-ba1b-4a10-a354-0087daf8d35a","Type":"ContainerStarted","Data":"db2b896f4ab9388dcae28c7814c138785a6cd60c6e746a501b1716102a83ced8"}
Nov 03 12:04:46.137078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.136669    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:46.138277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.138214    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59" event={"ID":"a33009df-f592-4370-b7e0-caea97754eed","Type":"ContainerStarted","Data":"442eece0863d7026e365e08879edcdb2ca4b12c4c2df56cd85c14b1b7913bdc7"}
Nov 03 12:04:46.138277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.138258    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59" event={"ID":"a33009df-f592-4370-b7e0-caea97754eed","Type":"ContainerStarted","Data":"7f946d378759da4a8556c33e0e2edcf7c83274a346d277dfad406563ef58e130"}
Nov 03 12:04:46.138412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.138284    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:46.242590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.242536    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" podStartSLOduration=1.242520464 podStartE2EDuration="1.242520464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:46.24198188 +0000 UTC m=+827.813565802" watchObservedRunningTime="2026-11-03 12:04:46.242520464 +0000 UTC m=+827.814104376"
Nov 03 12:04:46.349426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:46.349374    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59" podStartSLOduration=2.349356856 podStartE2EDuration="2.349356856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:46.347864062 +0000 UTC m=+827.919448066" watchObservedRunningTime="2026-11-03 12:04:46.349356856 +0000 UTC m=+827.920940768"
Nov 03 12:04:47.368456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.368422    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"]
Nov 03 12:04:47.372187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.372166    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.423729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.423693    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9893f0\""
Nov 03 12:04:47.423980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.423874    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9893f0-dockercfg-tncgr\""
Nov 03 12:04:47.477539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.477493    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"]
Nov 03 12:04:47.484721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.484687    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.484940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.484823    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tlt\" (UniqueName: \"kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt\") pod \"payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.529553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.529518    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"]
Nov 03 12:04:47.533371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.533349    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.586364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.586332    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tlt\" (UniqueName: \"kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt\") pod \"payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.586575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.586393    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.586575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.586423    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4ns\" (UniqueName: \"kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns\") pod \"payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.586707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.586687    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.590885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.590703    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"]
Nov 03 12:04:47.590885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.590868    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.642823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.642712    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tlt\" (UniqueName: \"kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt\") pod \"payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.685805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.685767    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:47.688253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.688221    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4ns\" (UniqueName: \"kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns\") pod \"payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.688414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.688302    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.689097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.688955    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.741792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.741763    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4ns\" (UniqueName: \"kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns\") pod \"payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:47.847012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:47.846981    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:48.047281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.047255    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"]
Nov 03 12:04:48.049114 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:48.049087    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05da420e_d1f6_446c_8256_0ee3bf572eae.slice/crio-39f8bc1a5c382bb70cbdc682934e9f34c47c61d49a3733cafb6d7e25a3959bf0 WatchSource:0}: Error finding container 39f8bc1a5c382bb70cbdc682934e9f34c47c61d49a3733cafb6d7e25a3959bf0: Status 404 returned error can't find the container with id 39f8bc1a5c382bb70cbdc682934e9f34c47c61d49a3733cafb6d7e25a3959bf0
Nov 03 12:04:48.150435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.150332    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" event={"ID":"05da420e-d1f6-446c-8256-0ee3bf572eae","Type":"ContainerStarted","Data":"f59b22a269c85edb2b1b6b4cec0a9d032ffa317ac230c767cf81b87148ae7d19"}
Nov 03 12:04:48.150435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.150402    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" event={"ID":"05da420e-d1f6-446c-8256-0ee3bf572eae","Type":"ContainerStarted","Data":"39f8bc1a5c382bb70cbdc682934e9f34c47c61d49a3733cafb6d7e25a3959bf0"}
Nov 03 12:04:48.150435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.150423    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:48.211211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.211186    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"]
Nov 03 12:04:48.213547 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:48.213520    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28468bae_eedf_4f25_a758_18a876eed1fc.slice/crio-7168f955a3b89e9f6ac36c7ca3f7fee5f4aafb8c1ccd4f09a4d0aa49256080da WatchSource:0}: Error finding container 7168f955a3b89e9f6ac36c7ca3f7fee5f4aafb8c1ccd4f09a4d0aa49256080da: Status 404 returned error can't find the container with id 7168f955a3b89e9f6ac36c7ca3f7fee5f4aafb8c1ccd4f09a4d0aa49256080da
Nov 03 12:04:48.256820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.256768    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" podStartSLOduration=1.256737678 podStartE2EDuration="1.256737678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:48.255219328 +0000 UTC m=+829.826803261" watchObservedRunningTime="2026-11-03 12:04:48.256737678 +0000 UTC m=+829.828321589"
Nov 03 12:04:48.937799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:48.937709    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:04:49.002348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002313    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002392    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002421    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002489    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002528    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002559    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002607    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002656    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:49.002715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002707    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002743    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket\") pod \"449e7deb-c747-49e0-a99b-be7a0041bcb3\" (UID: \"449e7deb-c747-49e0-a99b-be7a0041bcb3\") "
Nov 03 12:04:49.002941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002856    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:49.002941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002881    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:49.003079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.002940    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:49.003228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.003180    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:49.003228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.003202    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.003228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.003222    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.003228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.003232    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7deb-c747-49e0-a99b-be7a0041bcb3-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.003228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.003240    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.004984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.004955    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:04:49.005121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.004958    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:49.005385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.005369    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "kube-api-access-79gb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:49.005471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.005451    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo" pod "449e7deb-c747-49e0-a99b-be7a0041bcb3" (UID: "449e7deb-c747-49e0-a99b-be7a0041bcb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:04:49.104238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.104202    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gb9\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-kube-api-access-79gb9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.104238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.104232    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.104238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.104243    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.104551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.104253    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.104551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.104261    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7deb-c747-49e0-a99b-be7a0041bcb3-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:49.158595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.158561    4546 generic.go:356] "Generic (PLEG): container finished" podID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerID="e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61" exitCode=0
Nov 03 12:04:49.158807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.158620    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"
Nov 03 12:04:49.158807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.158653    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" event={"ID":"449e7deb-c747-49e0-a99b-be7a0041bcb3","Type":"ContainerDied","Data":"e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61"}
Nov 03 12:04:49.158807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.158711    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj" event={"ID":"449e7deb-c747-49e0-a99b-be7a0041bcb3","Type":"ContainerDied","Data":"eaf843dfc42707450049a562a96f3f4988cbe185ea5ff149814038c4b1c13c80"}
Nov 03 12:04:49.158807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.158736    4546 scope.go:122] "RemoveContainer" containerID="e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61"
Nov 03 12:04:49.160449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.160422    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" event={"ID":"28468bae-eedf-4f25-a758-18a876eed1fc","Type":"ContainerStarted","Data":"30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85"}
Nov 03 12:04:49.160576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.160475    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" event={"ID":"28468bae-eedf-4f25-a758-18a876eed1fc","Type":"ContainerStarted","Data":"7168f955a3b89e9f6ac36c7ca3f7fee5f4aafb8c1ccd4f09a4d0aa49256080da"}
Nov 03 12:04:49.160576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.160508    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:04:49.168951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.168935    4546 scope.go:122] "RemoveContainer" containerID="ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"
Nov 03 12:04:49.179144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.179123    4546 scope.go:122] "RemoveContainer" containerID="e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61"
Nov 03 12:04:49.179446 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:49.179429    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61\": container with ID starting with e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61 not found: ID does not exist" containerID="e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61"
Nov 03 12:04:49.179488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.179456    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61"} err="failed to get container status \"e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61\": rpc error: code = NotFound desc = could not find container \"e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61\": container with ID starting with e4cac4ec61ba9c7b9dbf9a6d273d84825a78bc5830e4dbf082473f8a6db66c61 not found: ID does not exist"
Nov 03 12:04:49.179488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.179474    4546 scope.go:122] "RemoveContainer" containerID="ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"
Nov 03 12:04:49.179720 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:04:49.179697    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2\": container with ID starting with ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2 not found: ID does not exist" containerID="ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"
Nov 03 12:04:49.179970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.179725    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2"} err="failed to get container status \"ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2\": rpc error: code = NotFound desc = could not find container \"ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2\": container with ID starting with ba8ccf05d50fc315ebe036fa270546670dc9fa273e8d493b9833bf280e0222c2 not found: ID does not exist"
Nov 03 12:04:49.311545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.311506    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"]
Nov 03 12:04:49.316797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.316768    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9ab28c7a-openshift-default-889fc7f49-npzgj"]
Nov 03 12:04:49.424860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:49.424804    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" podStartSLOduration=2.424787317 podStartE2EDuration="2.424787317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:49.422283761 +0000 UTC m=+830.993867694" watchObservedRunningTime="2026-11-03 12:04:49.424787317 +0000 UTC m=+830.996371238"
Nov 03 12:04:50.572278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:50.572243    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" path="/var/lib/kubelet/pods/449e7deb-c747-49e0-a99b-be7a0041bcb3/volumes"
Nov 03 12:04:55.131647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.131606    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"]
Nov 03 12:04:55.132381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.132362    4546 cpu_manager.go:414] "containerMap: removing container" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy"
Nov 03 12:04:55.132458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.132381    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy"
Nov 03 12:04:55.132458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.132395    4546 cpu_manager.go:414] "containerMap: removing container" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy"
Nov 03 12:04:55.132458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.132404    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy"
Nov 03 12:04:55.132567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.132525    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy"
Nov 03 12:04:55.137436 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.137416    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.190994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.190823    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"]
Nov 03 12:04:55.234514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.234485    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"kube-root-ca.crt\""
Nov 03 12:04:55.234514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.234506    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"openshift-service-ca.crt\""
Nov 03 12:04:55.234514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.234525    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"default-dockercfg-c46cq\""
Nov 03 12:04:55.234853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.234540    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"simulator-w2-91cd99-kserve-self-signed-certs\""
Nov 03 12:04:55.262264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.262223    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.262438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.262271    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.262438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.262397    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.262515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.262450    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvpw4\" (UniqueName: \"kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.262550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.262513    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.262583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.262554    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.363797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.363742    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.363797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.363800    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.363860    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.363912    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvpw4\" (UniqueName: \"kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.363941    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.363960    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.364396    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.364421    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.364588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.364456    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.366277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.366258    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.366431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.366414    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.424721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.424641    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvpw4\" (UniqueName: \"kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4\") pod \"simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") " pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.452525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.452489    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:55.830779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:55.829377    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"]
Nov 03 12:04:55.837386 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:55.837350    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbdf272c_f4ab_413c_b83d_9d6aacdd4bc6.slice/crio-f4c68abc922f80511ea7173f330dee19faed425dcb66a33fde17d582fbec9b8d WatchSource:0}: Error finding container f4c68abc922f80511ea7173f330dee19faed425dcb66a33fde17d582fbec9b8d: Status 404 returned error can't find the container with id f4c68abc922f80511ea7173f330dee19faed425dcb66a33fde17d582fbec9b8d
Nov 03 12:04:56.234175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:56.234137    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" event={"ID":"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6","Type":"ContainerStarted","Data":"e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090"}
Nov 03 12:04:56.234610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:56.234188    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" event={"ID":"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6","Type":"ContainerStarted","Data":"f4c68abc922f80511ea7173f330dee19faed425dcb66a33fde17d582fbec9b8d"}
Nov 03 12:04:56.342432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:56.342371    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podStartSLOduration=1.342348555 podStartE2EDuration="1.342348555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:56.339888485 +0000 UTC m=+837.911472397" watchObservedRunningTime="2026-11-03 12:04:56.342348555 +0000 UTC m=+837.913932472"
Nov 03 12:04:56.453527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:56.453477    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:04:57.146540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:57.146511    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 12:04:57.146696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:57.146578    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 12:04:58.504368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.504329    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"]
Nov 03 12:04:58.505142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.505121    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="449e7deb-c747-49e0-a99b-be7a0041bcb3" containerName="istio-proxy"
Nov 03 12:04:58.510774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.509272    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.603852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.603820    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bg28\" (UniqueName: \"kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28\") pod \"payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.604017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.603885    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.605227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.605199    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5951bc\""
Nov 03 12:04:58.605398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.605371    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5951bc-dockercfg-lffgb\""
Nov 03 12:04:58.614780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.607966    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"]
Nov 03 12:04:58.704774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.704725    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bg28\" (UniqueName: \"kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28\") pod \"payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.704882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.704849    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.705500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.705471    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.757520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.757446    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bg28\" (UniqueName: \"kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28\") pod \"payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:58.845719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:58.845678    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:04:59.164254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:59.164176    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 12:04:59.245611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:04:59.245588    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"]
Nov 03 12:04:59.248250 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:04:59.248223    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7207c66b_8ae9_491f_b723_9cbfe619fca6.slice/crio-99697b07294e81c33f557b298f4204a4d215435bfc1e50f0d72d191be1f79184 WatchSource:0}: Error finding container 99697b07294e81c33f557b298f4204a4d215435bfc1e50f0d72d191be1f79184: Status 404 returned error can't find the container with id 99697b07294e81c33f557b298f4204a4d215435bfc1e50f0d72d191be1f79184
Nov 03 12:05:00.179324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:00.179290    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 12:05:00.257985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:00.257951    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v" event={"ID":"7207c66b-8ae9-491f-b723-9cbfe619fca6","Type":"ContainerStarted","Data":"c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f"}
Nov 03 12:05:00.258162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:00.258009    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v" event={"ID":"7207c66b-8ae9-491f-b723-9cbfe619fca6","Type":"ContainerStarted","Data":"99697b07294e81c33f557b298f4204a4d215435bfc1e50f0d72d191be1f79184"}
Nov 03 12:05:00.258162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:00.258038    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:05:00.392591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:00.392539    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v" podStartSLOduration=2.392524278 podStartE2EDuration="2.392524278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:00.392204855 +0000 UTC m=+841.963788778" watchObservedRunningTime="2026-11-03 12:05:00.392524278 +0000 UTC m=+841.964108189"
Nov 03 12:05:03.409064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.409025    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"]
Nov 03 12:05:03.412952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.412931    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.511600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.511564    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"]
Nov 03 12:05:03.585652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.585618    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.585867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.585661    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.585867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.585693    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.585867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.585799    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.585867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.585857    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.586071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.585904    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sq5l\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.586071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.586010    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.586157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.586124    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.586204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.586173    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.686988    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687046    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687073    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687090    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687125    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687166    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687195    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687228    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687285    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq5l\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687588    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687791    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687872    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.687944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.687894    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.688045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.688029    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.689505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.689482    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.690117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.690100    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.741501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.741463    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:03.742390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:03.742360    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sq5l\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:04.029163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.029137    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:04.289343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.289274    4546 generic.go:356] "Generic (PLEG): container finished" podID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerID="37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae" exitCode=0
Nov 03 12:05:04.289483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.289341    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" event={"ID":"9b43c591-0ba5-419f-8bf0-55c164c532e2","Type":"ContainerDied","Data":"37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"}
Nov 03 12:05:04.289738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.289723    4546 scope.go:122] "RemoveContainer" containerID="37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"
Nov 03 12:05:04.291078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.291047    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:04.291192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.291135    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:04.291192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.291181    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:04.447358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.447328    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"]
Nov 03 12:05:04.451269 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:05:04.451244    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381ed6a4_0fd7_47e4_95a8_9a6c119b3743.slice/crio-efdd979a977acee7548664d8454e433c375e40e69aac6668b83cbef9a7936b53 WatchSource:0}: Error finding container efdd979a977acee7548664d8454e433c375e40e69aac6668b83cbef9a7936b53: Status 404 returned error can't find the container with id efdd979a977acee7548664d8454e433c375e40e69aac6668b83cbef9a7936b53
Nov 03 12:05:04.453826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.453743    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:04.453920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.453865    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:04.453920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:04.453895    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:05.068273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.068240    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:05:05.296500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.296463    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" event={"ID":"9b43c591-0ba5-419f-8bf0-55c164c532e2","Type":"ContainerStarted","Data":"aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614"}
Nov 03 12:05:05.298009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.297981    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" event={"ID":"381ed6a4-0fd7-47e4-95a8-9a6c119b3743","Type":"ContainerStarted","Data":"c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a"}
Nov 03 12:05:05.298132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.298032    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" event={"ID":"381ed6a4-0fd7-47e4-95a8-9a6c119b3743","Type":"ContainerStarted","Data":"efdd979a977acee7548664d8454e433c375e40e69aac6668b83cbef9a7936b53"}
Nov 03 12:05:05.347247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.347109    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:05:05.348881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.348843    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 12:05:05.349157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.349132    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 12:05:05.456033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.455991    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:05:05.471340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.471311    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:05:05.540784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:05.539467    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podStartSLOduration=2.539441317 podStartE2EDuration="2.539441317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:05.526134673 +0000 UTC m=+847.097718609" watchObservedRunningTime="2026-11-03 12:05:05.539441317 +0000 UTC m=+847.111025231"
Nov 03 12:05:06.037775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:06.037564    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:06.057778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:06.052891    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:05:06.057778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:06.052952    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 12:05:06.358575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:06.358478    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 12:05:06.358575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:06.358542    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 12:05:06.363560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:06.363535    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:05:07.030513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:07.030070    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:05:07.030513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:07.030133    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 12:05:07.357834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:07.357272    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 12:05:07.357834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:07.357336    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 12:05:07.679795 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:05:07.679670    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:05:08.038774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:08.038220    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:05:08.038774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:08.038282    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 12:05:08.347637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:08.347551    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 12:05:08.347934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:08.347905    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 12:05:09.031777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:09.031244    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:05:09.031777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:09.031308    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 12:05:09.349832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:09.349133    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 12:05:09.349832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:09.349194    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 12:05:10.031775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:10.030792    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 12:05:10.031775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:10.030855    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 12:05:10.341470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:10.341378    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 12:05:10.341470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:10.341447    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 12:05:11.033980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.033949    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:11.034444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.034364    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:11.035085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.035065    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:05:11.266942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.266915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 12:05:11.345541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.345467    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:05:11.345811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.345795    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:05:11.346611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:11.346591    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:05:20.677888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.677852    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"]
Nov 03 12:05:20.682874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.682843    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714309    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzgx\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714350    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714371    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714449    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714589    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714634    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714795    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714862    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.714954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.714932    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816188    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzgx\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816233    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816251    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816291    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816336    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816365    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816531    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816562    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.816964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816936    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.817034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.816936    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.817034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.817014    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.817149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.817068    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.817206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.817169    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.819721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.819697    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.820050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.820027    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.878367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.878281    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"]
Nov 03 12:05:20.881724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.881340    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzgx\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.883648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.883597    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:20.996907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:20.996871    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:21.458112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:21.458086    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"]
Nov 03 12:05:21.459937 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:05:21.459871    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0e3ad4_23fd_4535_bef5_a2a475c52d95.slice/crio-919789aa1385c2b56e088ddee01263710128473afc4e195d8c3705fe1666d10a WatchSource:0}: Error finding container 919789aa1385c2b56e088ddee01263710128473afc4e195d8c3705fe1666d10a: Status 404 returned error can't find the container with id 919789aa1385c2b56e088ddee01263710128473afc4e195d8c3705fe1666d10a
Nov 03 12:05:21.462374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:21.462343    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:21.462454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:21.462408    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:21.462454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:21.462440    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:22.448125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:22.448080    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" event={"ID":"dc0e3ad4-23fd-4535-bef5-a2a475c52d95","Type":"ContainerStarted","Data":"291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83"}
Nov 03 12:05:22.448553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:22.448162    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" event={"ID":"dc0e3ad4-23fd-4535-bef5-a2a475c52d95","Type":"ContainerStarted","Data":"919789aa1385c2b56e088ddee01263710128473afc4e195d8c3705fe1666d10a"}
Nov 03 12:05:22.603249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:22.603195    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podStartSLOduration=2.603180629 podStartE2EDuration="2.603180629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:22.602027725 +0000 UTC m=+864.173611654" watchObservedRunningTime="2026-11-03 12:05:22.603180629 +0000 UTC m=+864.174764539"
Nov 03 12:05:22.997271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:22.997186    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:22.998939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:22.998915    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 12:05:22.999054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:22.998960    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 12:05:23.091916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.091879    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:23.092079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.091928    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:23.454505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.454478    4546 generic.go:356] "Generic (PLEG): container finished" podID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerID="f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec" exitCode=0
Nov 03 12:05:23.454965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.454560    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" event={"ID":"511d427a-a19d-411f-9c7c-2fd4674f2c73","Type":"ContainerDied","Data":"f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"}
Nov 03 12:05:23.455062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.455047    4546 scope.go:122] "RemoveContainer" containerID="f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"
Nov 03 12:05:23.456240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.456216    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:23.456301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.456279    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:23.456336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.456308    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:23.997644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.997606    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 12:05:23.997891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:23.997691    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 12:05:24.469399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:24.469359    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" event={"ID":"511d427a-a19d-411f-9c7c-2fd4674f2c73","Type":"ContainerStarted","Data":"602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80"}
Nov 03 12:05:24.553275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:24.553239    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:05:24.553649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:24.553624    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:24.553843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:24.553818    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:24.997623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:24.997587    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 12:05:24.997815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:24.997641    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 12:05:25.554779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:25.554159    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:25.554779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:25.554225    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:25.998183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:25.998148    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 12:05:25.998372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:25.998203    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 12:05:26.553284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:26.553248    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:26.553706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:26.553663    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:26.997406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:26.997369    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 12:05:26.997833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:26.997426    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 12:05:27.552005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:27.551962    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:27.552194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:27.552031    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:28.002572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.002539    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:28.003018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.002810    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:28.003533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.003514    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:05:28.098939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.098902    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"]
Nov 03 12:05:28.102814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.102791    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.152221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.152186    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"kube-root-ca.crt\""
Nov 03 12:05:28.152390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.152248    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"default-dockercfg-n4q48\""
Nov 03 12:05:28.153557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.153535    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"]
Nov 03 12:05:28.203095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.203058    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.203251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.203185    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.203297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.203248    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9brn\" (UniqueName: \"kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.203297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.203281    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.203366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.203349    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.203398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.203365    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.251889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.251856    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"simulator-w4-8d2c7c-kserve-self-signed-certs\""
Nov 03 12:05:28.251889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.251856    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"openshift-service-ca.crt\""
Nov 03 12:05:28.304385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304292    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.304546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304408    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.304546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304493    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9brn\" (UniqueName: \"kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.304663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304541    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.304663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304611    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.304663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304636    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.304926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.304867    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.305294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.305272    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.305671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.305645    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.338055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.338029    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.338342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.338315    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.359524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.359487    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9brn\" (UniqueName: \"kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn\") pod \"simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") " pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.440130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.440089    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:28.552209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.552170    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:28.552522 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.552231    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:28.806391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:28.806368    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"]
Nov 03 12:05:29.497948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:29.497908    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" event={"ID":"d8c714d2-b2d5-4f50-a3f6-5972844d1678","Type":"ContainerStarted","Data":"6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b"}
Nov 03 12:05:29.498358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:29.497958    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" event={"ID":"d8c714d2-b2d5-4f50-a3f6-5972844d1678","Type":"ContainerStarted","Data":"eeb1d6d7b54784b434dec1d834e2c8c3dc883fec448840da48fbc15d98e6bf24"}
Nov 03 12:05:29.551800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:29.551759    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 12:05:29.552013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:29.551834    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 12:05:29.605411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:29.605356    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podStartSLOduration=1.60534058 podStartE2EDuration="1.60534058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:29.604164453 +0000 UTC m=+871.175748365" watchObservedRunningTime="2026-11-03 12:05:29.60534058 +0000 UTC m=+871.176924491"
Nov 03 12:05:30.440580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:30.440543    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:30.557307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:30.557274    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:05:30.557743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:30.557634    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:05:30.558303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:30.558285    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:05:37.653947 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:05:37.653904    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:05:38.441193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:38.441163    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:38.453643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:38.453609    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:38.571733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:38.571700    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:05:48.688089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.688052    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"]
Nov 03 12:05:48.691871 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.691851    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776226    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4r4\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776301    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776371    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776585 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776480    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776585 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776522    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776615    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776679    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776785    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.776838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.776815    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.789554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.789520    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"]
Nov 03 12:05:48.877504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4r4\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.877708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877517    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.877708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877574    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.877708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877612    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.877708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877637    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.877708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877681    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877742    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877799    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.877821    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.878078    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.878125    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.878303    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.878334    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.878712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.878630    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.880388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.880367    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.880690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.880671    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.930988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.930956    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:48.931137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:48.931100    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4r4\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:49.005762 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.005713    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:49.418210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.418187    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"]
Nov 03 12:05:49.419850 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:05:49.419822    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70d5924_b47a_43dd_b583_0a2e381350ee.slice/crio-34bb8cb39aa69e17bcdf65b5b31015b8ffa0aed6944d76e50c4ef0199225d853 WatchSource:0}: Error finding container 34bb8cb39aa69e17bcdf65b5b31015b8ffa0aed6944d76e50c4ef0199225d853: Status 404 returned error can't find the container with id 34bb8cb39aa69e17bcdf65b5b31015b8ffa0aed6944d76e50c4ef0199225d853
Nov 03 12:05:49.422292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.422264    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:49.422419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.422349    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:49.422419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.422397    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:05:49.617018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.616981    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" event={"ID":"b70d5924-b47a-43dd-b583-0a2e381350ee","Type":"ContainerStarted","Data":"2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd"}
Nov 03 12:05:49.617214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.617029    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" event={"ID":"b70d5924-b47a-43dd-b583-0a2e381350ee","Type":"ContainerStarted","Data":"34bb8cb39aa69e17bcdf65b5b31015b8ffa0aed6944d76e50c4ef0199225d853"}
Nov 03 12:05:49.725882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:49.725834    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podStartSLOduration=1.725819134 podStartE2EDuration="1.725819134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:49.7251985 +0000 UTC m=+891.296782475" watchObservedRunningTime="2026-11-03 12:05:49.725819134 +0000 UTC m=+891.297403046"
Nov 03 12:05:50.006535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:50.006450    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:50.008376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:50.008337    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:05:50.008685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:50.008652    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 12:05:51.028779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:51.028328    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:05:51.028779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:51.028401    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 12:05:52.006313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:52.006279    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:05:52.006497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:52.006333    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 12:05:53.006594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.006555    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:05:53.007103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.006610    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 12:05:53.866455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.866413    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"]
Nov 03 12:05:53.870354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.870331    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.947241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.947198    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.947422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.947395    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.947472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.947445    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.947543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.947527    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.947616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.947604    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.947712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.947689    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrg2\" (UniqueName: \"kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:53.979803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.979742    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"default-dockercfg-thlkn\""
Nov 03 12:05:53.979956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.979843    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"simulator-w0-9893f0-kserve-self-signed-certs\""
Nov 03 12:05:53.980002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.979964    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"openshift-service-ca.crt\""
Nov 03 12:05:53.980002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.979967    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"kube-root-ca.crt\""
Nov 03 12:05:53.982204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:53.982186    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"]
Nov 03 12:05:54.006573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.006547    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:05:54.006691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.006588    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 12:05:54.049134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049092    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.049355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049168    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrg2\" (UniqueName: \"kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.049355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049211    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.049355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049301    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.049355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049332    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.049573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049383    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.049870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049845    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.050076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.049948    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.050240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.050220    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.052283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.052262    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.052681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.052667    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.132563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.132479    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrg2\" (UniqueName: \"kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2\") pod \"simulator-w0-9893f0-kserve-684c984b74-nnl68\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") " pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.182665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.182637    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:05:54.573125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.573099    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"]
Nov 03 12:05:54.574286 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:05:54.574256    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4e4feb2_507e_44a8_b9e0_5ac70b9a3450.slice/crio-aa639fee78e662ea72e5f147be7e1e7373d6d93b3bc85530d037c2a4b75bfc7f WatchSource:0}: Error finding container aa639fee78e662ea72e5f147be7e1e7373d6d93b3bc85530d037c2a4b75bfc7f: Status 404 returned error can't find the container with id aa639fee78e662ea72e5f147be7e1e7373d6d93b3bc85530d037c2a4b75bfc7f
Nov 03 12:05:54.657991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:54.657956    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" event={"ID":"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450","Type":"ContainerStarted","Data":"aa639fee78e662ea72e5f147be7e1e7373d6d93b3bc85530d037c2a4b75bfc7f"}
Nov 03 12:05:55.006420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:55.006390    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 12:05:55.006630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:55.006437    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 12:05:55.666390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:55.666356    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" event={"ID":"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450","Type":"ContainerStarted","Data":"daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d"}
Nov 03 12:05:55.774842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:55.774793    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podStartSLOduration=2.774777703 podStartE2EDuration="2.774777703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:55.772853091 +0000 UTC m=+897.344437025" watchObservedRunningTime="2026-11-03 12:05:55.774777703 +0000 UTC m=+897.346361615"
Nov 03 12:05:56.010152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:56.010125    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:56.010533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:56.010508    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:56.011201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:56.011185    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:05:56.182814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:05:56.182776    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:06:00.928065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:00.928031    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"]
Nov 03 12:06:00.933382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:00.933359    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:00.985007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:00.984975    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91cd99-openshift-default-dockercfg-pj65x\""
Nov 03 12:06:01.017313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017281    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017335    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017431    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcqn4\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017464    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017487    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017502    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017678    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017744    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.017928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.017844    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.034018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.033992    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"]
Nov 03 12:06:01.118619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118590    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118622    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118652    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118675    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118712    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118742    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcqn4\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118798    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118833    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.118916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.118858    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.119240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.119080    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.119240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.119140    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.119307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.119239    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.119307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.119269    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.119406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.119383    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.121405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.121381    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.121597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.121580    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.172128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.172102    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.172447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.172422    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcqn4\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.246523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.246487    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:01.660867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.660842    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"]
Nov 03 12:06:01.662516 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:06:01.662488    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda884db75_8303_42ab_8a47_d94fda05a24c.slice/crio-e8ab89d9a2a63c8f4d5aa48f8d5a479cdbc85eef14f3434a6499330183a31d1b WatchSource:0}: Error finding container e8ab89d9a2a63c8f4d5aa48f8d5a479cdbc85eef14f3434a6499330183a31d1b: Status 404 returned error can't find the container with id e8ab89d9a2a63c8f4d5aa48f8d5a479cdbc85eef14f3434a6499330183a31d1b
Nov 03 12:06:01.665042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.665011    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:06:01.665132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.665084    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:06:01.665174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.665146    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:06:01.699181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:01.699151    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" event={"ID":"a884db75-8303-42ab-8a47-d94fda05a24c","Type":"ContainerStarted","Data":"e8ab89d9a2a63c8f4d5aa48f8d5a479cdbc85eef14f3434a6499330183a31d1b"}
Nov 03 12:06:02.706307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:02.706273    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" event={"ID":"a884db75-8303-42ab-8a47-d94fda05a24c","Type":"ContainerStarted","Data":"e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa"}
Nov 03 12:06:02.865184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:02.865114    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podStartSLOduration=2.8651003250000002 podStartE2EDuration="2.865100325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:02.86203298 +0000 UTC m=+904.433616889" watchObservedRunningTime="2026-11-03 12:06:02.865100325 +0000 UTC m=+904.436684236"
Nov 03 12:06:03.247074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:03.247032    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:03.248952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:03.248917    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:06:03.249123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:03.249018    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 12:06:04.183062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:04.183028    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:06:04.195629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:04.195607    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:06:04.247312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:04.247277    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:06:04.247527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:04.247346    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 12:06:04.750105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:04.750078    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:06:05.247588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:05.247556    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:06:05.248006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:05.247608    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 12:06:06.247656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:06.247615    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:06:06.248185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:06.247676    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 12:06:07.247557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:07.247518    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:06:07.247722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:07.247583    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 12:06:07.650570 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:06:07.650476    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:06:08.336792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:08.336144    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:08.336792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:08.336565    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:08.337556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:08.337535    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:06:36.769471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.769438    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"]
Nov 03 12:06:36.774461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.774438    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.819779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.819714    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99r6c\" (UniqueName: \"kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.819978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.819843    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.819978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.819920    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.871108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.871039    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:06:36.871108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.871077    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:06:36.871334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.871078    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:06:36.871334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.871116    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:06:36.873476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.873456    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"]
Nov 03 12:06:36.921433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.921398    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.921645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.921521    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99r6c\" (UniqueName: \"kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.921645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.921552    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.922337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.922316    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.924016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.923995    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:36.975439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:36.975407    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99r6c\" (UniqueName: \"kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c\") pod \"maas-api-revoke-sub-e58bc805-4fqz2\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:37.088866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:37.088786    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:37.453981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:37.453958    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"]
Nov 03 12:06:37.455830 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:06:37.455805    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c959724_1374_4817_ac2e_e32121f9a7f5.slice/crio-87b2989b6ea55c2ba8397287e331b01dcee9944a75bd897889d471d618ca0a30 WatchSource:0}: Error finding container 87b2989b6ea55c2ba8397287e331b01dcee9944a75bd897889d471d618ca0a30: Status 404 returned error can't find the container with id 87b2989b6ea55c2ba8397287e331b01dcee9944a75bd897889d471d618ca0a30
Nov 03 12:06:37.650539 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:06:37.650508    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:06:37.919158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:37.919126    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2" event={"ID":"5c959724-1374-4817-ac2e-e32121f9a7f5","Type":"ContainerStarted","Data":"f95fcd89d7c11b296c7b7e70fc39121e36100b09d8771b4fb7abce46e03209db"}
Nov 03 12:06:37.919554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:37.919189    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2" event={"ID":"5c959724-1374-4817-ac2e-e32121f9a7f5","Type":"ContainerStarted","Data":"87b2989b6ea55c2ba8397287e331b01dcee9944a75bd897889d471d618ca0a30"}
Nov 03 12:06:38.027497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:38.027401    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2" podStartSLOduration=2.027387169 podStartE2EDuration="2.027387169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:38.023955241 +0000 UTC m=+939.595539153" watchObservedRunningTime="2026-11-03 12:06:38.027387169 +0000 UTC m=+939.598971080"
Nov 03 12:06:38.982063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:38.982019    4546 generic.go:356] "Generic (PLEG): container finished" podID="5c959724-1374-4817-ac2e-e32121f9a7f5" containerID="f95fcd89d7c11b296c7b7e70fc39121e36100b09d8771b4fb7abce46e03209db" exitCode=0
Nov 03 12:06:38.983258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:38.983076    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2" event={"ID":"5c959724-1374-4817-ac2e-e32121f9a7f5","Type":"ContainerDied","Data":"f95fcd89d7c11b296c7b7e70fc39121e36100b09d8771b4fb7abce46e03209db"}
Nov 03 12:06:40.137925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.137896    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:40.236563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.236533    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca\") pod \"5c959724-1374-4817-ac2e-e32121f9a7f5\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") "
Nov 03 12:06:40.236715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.236689    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca\") pod \"5c959724-1374-4817-ac2e-e32121f9a7f5\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") "
Nov 03 12:06:40.236788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.236735    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c\" (UniqueName: \"kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c\") pod \"5c959724-1374-4817-ac2e-e32121f9a7f5\" (UID: \"5c959724-1374-4817-ac2e-e32121f9a7f5\") "
Nov 03 12:06:40.237067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.237032    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca" pod "5c959724-1374-4817-ac2e-e32121f9a7f5" (UID: "5c959724-1374-4817-ac2e-e32121f9a7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:06:40.237189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.237170    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:06:40.238962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.238937    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c" pod "5c959724-1374-4817-ac2e-e32121f9a7f5" (UID: "5c959724-1374-4817-ac2e-e32121f9a7f5"). InnerVolumeSpecName "kube-api-access-99r6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:06:40.239096 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.239057    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca" pod "5c959724-1374-4817-ac2e-e32121f9a7f5" (UID: "5c959724-1374-4817-ac2e-e32121f9a7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:06:40.338300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.338266    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c959724-1374-4817-ac2e-e32121f9a7f5-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:06:40.338300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.338296    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99r6c\" (UniqueName: \"kubernetes.io/projected/5c959724-1374-4817-ac2e-e32121f9a7f5-kube-api-access-99r6c\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:06:40.995194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.995158    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"
Nov 03 12:06:40.995381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.995163    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2" event={"ID":"5c959724-1374-4817-ac2e-e32121f9a7f5","Type":"ContainerDied","Data":"87b2989b6ea55c2ba8397287e331b01dcee9944a75bd897889d471d618ca0a30"}
Nov 03 12:06:40.995381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:40.995284    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b2989b6ea55c2ba8397287e331b01dcee9944a75bd897889d471d618ca0a30"
Nov 03 12:06:42.137664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.137626    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"]
Nov 03 12:06:42.138225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.138208    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5c959724-1374-4817-ac2e-e32121f9a7f5" containerName="revoke-keys"
Nov 03 12:06:42.138279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.138227    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c959724-1374-4817-ac2e-e32121f9a7f5" containerName="revoke-keys"
Nov 03 12:06:42.138323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.138310    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c959724-1374-4817-ac2e-e32121f9a7f5" containerName="revoke-keys"
Nov 03 12:06:42.141554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.141537    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.155084 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.155058    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.155190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.155114    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.155244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.155226    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.155291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.155284    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.155328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.155299    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4sc\" (UniqueName: \"kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.155443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.155426    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.238859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.238827    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"premium-w0-9893f0-kserve-self-signed-certs\""
Nov 03 12:06:42.241085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.241033    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"]
Nov 03 12:06:42.256506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.256479    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.256652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.256535    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.256652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.256566    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.256652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.256594    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.256652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.256650    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.256903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.256674    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4sc\" (UniqueName: \"kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.257117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.257090    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.257242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.257171    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.257306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.257243    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.259067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.259044    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.259425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.259401    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.310192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.310162    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4sc\" (UniqueName: \"kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc\") pod \"premium-w0-9893f0-kserve-6779bb7f9-mg72m\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") " pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.452920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.452819    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:42.831707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:42.831680    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"]
Nov 03 12:06:42.833486 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:06:42.833455    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e30c5cb_3d9b_49da_af00_2c2ac2253a47.slice/crio-717387b3bdd7ef00e1e7b755f4b014fc0fc4c3a269ae814fc28c12db98985c7c WatchSource:0}: Error finding container 717387b3bdd7ef00e1e7b755f4b014fc0fc4c3a269ae814fc28c12db98985c7c: Status 404 returned error can't find the container with id 717387b3bdd7ef00e1e7b755f4b014fc0fc4c3a269ae814fc28c12db98985c7c
Nov 03 12:06:43.007643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:43.007606    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" event={"ID":"8e30c5cb-3d9b-49da-af00-2c2ac2253a47","Type":"ContainerStarted","Data":"66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6"}
Nov 03 12:06:43.007838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:43.007651    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" event={"ID":"8e30c5cb-3d9b-49da-af00-2c2ac2253a47","Type":"ContainerStarted","Data":"717387b3bdd7ef00e1e7b755f4b014fc0fc4c3a269ae814fc28c12db98985c7c"}
Nov 03 12:06:43.115594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:43.115488    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podStartSLOduration=1.115474132 podStartE2EDuration="1.115474132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:43.114275903 +0000 UTC m=+944.685859827" watchObservedRunningTime="2026-11-03 12:06:43.115474132 +0000 UTC m=+944.687058043"
Nov 03 12:06:43.453462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:43.453386    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:52.453301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:52.453264    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:52.473326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:52.473299    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:53.073696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:53.073671    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:06:54.685544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.685014    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"]
Nov 03 12:06:54.691225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.691199    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.744561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.744532    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"]
Nov 03 12:06:54.779941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.779910    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7sf\" (UniqueName: \"kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.780105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.779975    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.780159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.780096    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.788135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.788107    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:06:54.788135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.788120    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:06:54.788283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.788139    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:06:54.788283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.788171    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:06:54.881641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.881614    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7sf\" (UniqueName: \"kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.881803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.881679    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.881803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.881707    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.882328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.882308    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.884259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.884235    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:54.934592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:54.934566    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7sf\" (UniqueName: \"kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf\") pod \"maas-api-revoke-sub-5eab9b0a-jzvrw\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:55.006294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:55.006264    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:55.370617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:55.370594    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"]
Nov 03 12:06:55.372184 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:06:55.372156    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96729d7c_23f2_4cd2_915c_bdb0440bfaf3.slice/crio-f87689546bbcc5da6e1a8fa2b68e40519b908407a03b94e3a1a2273bc36a79a5 WatchSource:0}: Error finding container f87689546bbcc5da6e1a8fa2b68e40519b908407a03b94e3a1a2273bc36a79a5: Status 404 returned error can't find the container with id f87689546bbcc5da6e1a8fa2b68e40519b908407a03b94e3a1a2273bc36a79a5
Nov 03 12:06:56.092178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:56.092145    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw" event={"ID":"96729d7c-23f2-4cd2-915c-bdb0440bfaf3","Type":"ContainerStarted","Data":"8a4babf2944a98c2816a717dc585bf2131c44988958f00ea9c04d4612becfbf8"}
Nov 03 12:06:56.092178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:56.092194    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw" event={"ID":"96729d7c-23f2-4cd2-915c-bdb0440bfaf3","Type":"ContainerStarted","Data":"f87689546bbcc5da6e1a8fa2b68e40519b908407a03b94e3a1a2273bc36a79a5"}
Nov 03 12:06:56.197699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:56.197647    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw" podStartSLOduration=2.197634379 podStartE2EDuration="2.197634379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:56.19721421 +0000 UTC m=+957.768798150" watchObservedRunningTime="2026-11-03 12:06:56.197634379 +0000 UTC m=+957.769218290"
Nov 03 12:06:57.098833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:57.098797    4546 generic.go:356] "Generic (PLEG): container finished" podID="96729d7c-23f2-4cd2-915c-bdb0440bfaf3" containerID="8a4babf2944a98c2816a717dc585bf2131c44988958f00ea9c04d4612becfbf8" exitCode=0
Nov 03 12:06:57.099220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:57.098862    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw" event={"ID":"96729d7c-23f2-4cd2-915c-bdb0440bfaf3","Type":"ContainerDied","Data":"8a4babf2944a98c2816a717dc585bf2131c44988958f00ea9c04d4612becfbf8"}
Nov 03 12:06:58.263728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.263700    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:58.417375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.417285    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf\" (UniqueName: \"kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf\") pod \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") "
Nov 03 12:06:58.417375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.417362    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca\") pod \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") "
Nov 03 12:06:58.417619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.417407    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca\") pod \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\" (UID: \"96729d7c-23f2-4cd2-915c-bdb0440bfaf3\") "
Nov 03 12:06:58.417855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.417822    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca" pod "96729d7c-23f2-4cd2-915c-bdb0440bfaf3" (UID: "96729d7c-23f2-4cd2-915c-bdb0440bfaf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:06:58.420078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.420052    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca" pod "96729d7c-23f2-4cd2-915c-bdb0440bfaf3" (UID: "96729d7c-23f2-4cd2-915c-bdb0440bfaf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:06:58.420231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.420214    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf" pod "96729d7c-23f2-4cd2-915c-bdb0440bfaf3" (UID: "96729d7c-23f2-4cd2-915c-bdb0440bfaf3"). InnerVolumeSpecName "kube-api-access-2m7sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:06:58.519010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.518985    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m7sf\" (UniqueName: \"kubernetes.io/projected/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-kube-api-access-2m7sf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:06:58.519010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.519011    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:06:58.519188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:58.519021    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96729d7c-23f2-4cd2-915c-bdb0440bfaf3-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:06:59.110900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:59.110870    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"
Nov 03 12:06:59.111071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:59.110902    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw" event={"ID":"96729d7c-23f2-4cd2-915c-bdb0440bfaf3","Type":"ContainerDied","Data":"f87689546bbcc5da6e1a8fa2b68e40519b908407a03b94e3a1a2273bc36a79a5"}
Nov 03 12:06:59.111071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:06:59.110944    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f87689546bbcc5da6e1a8fa2b68e40519b908407a03b94e3a1a2273bc36a79a5"
Nov 03 12:07:04.903419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:04.903389    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"]
Nov 03 12:07:04.903832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:04.903613    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy" containerID="cri-o://d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652" gracePeriod=30
Nov 03 12:07:07.650459 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:07:07.650420    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:07:08.588589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.588558    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"]
Nov 03 12:07:08.589095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.589080    4546 cpu_manager.go:414] "containerMap: removing container" podUID="96729d7c-23f2-4cd2-915c-bdb0440bfaf3" containerName="revoke-keys"
Nov 03 12:07:08.589151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.589097    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96729d7c-23f2-4cd2-915c-bdb0440bfaf3" containerName="revoke-keys"
Nov 03 12:07:08.589199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.589189    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="96729d7c-23f2-4cd2-915c-bdb0440bfaf3" containerName="revoke-keys"
Nov 03 12:07:08.593574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.593552    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.641143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.641113    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.641345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.641179    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.641345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.641296    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.641345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.641329    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfltj\" (UniqueName: \"kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.641517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.641451    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.641517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.641488    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.690445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.690413    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"kube-root-ca.crt\""
Nov 03 12:07:08.691009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.690480    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"openshift-service-ca.crt\""
Nov 03 12:07:08.691009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.690611    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"premium-w3-bb1b51-kserve-self-signed-certs\""
Nov 03 12:07:08.691009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.690653    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"default-dockercfg-87hfd\""
Nov 03 12:07:08.693348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.693320    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"]
Nov 03 12:07:08.742147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742115    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfltj\" (UniqueName: \"kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742194    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742225    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742304    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742410    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742474    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742715    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742782    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.742883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.742851    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.745035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.744999    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.745142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.745096    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.795718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.795686    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfltj\" (UniqueName: \"kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj\") pod \"premium-w3-bb1b51-kserve-869dc8dd48-brhft\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") " pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:08.905788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:08.905664    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:09.298452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:09.298429    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"]
Nov 03 12:07:09.300045 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:07:09.300016    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a4b8978_01ca_4005_99e5_fc3cbd4edb59.slice/crio-c8ca24a4e136106c727f411d94a8ad5a400990ad1868db58e3be6934b5b7e380 WatchSource:0}: Error finding container c8ca24a4e136106c727f411d94a8ad5a400990ad1868db58e3be6934b5b7e380: Status 404 returned error can't find the container with id c8ca24a4e136106c727f411d94a8ad5a400990ad1868db58e3be6934b5b7e380
Nov 03 12:07:10.166705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.166654    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:07:10.178225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.178203    4546 generic.go:356] "Generic (PLEG): container finished" podID="07785463-7365-4b0b-9ece-ebefa51c7469" containerID="d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652" exitCode=0
Nov 03 12:07:10.178348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.178243    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"
Nov 03 12:07:10.178348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.178289    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" event={"ID":"07785463-7365-4b0b-9ece-ebefa51c7469","Type":"ContainerDied","Data":"d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652"}
Nov 03 12:07:10.178348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.178342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq" event={"ID":"07785463-7365-4b0b-9ece-ebefa51c7469","Type":"ContainerDied","Data":"bf71317af60d872288b1e31d90493d46a0a6cbb188554aeae20c5f68a3fc9ba5"}
Nov 03 12:07:10.178509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.178371    4546 scope.go:122] "RemoveContainer" containerID="d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652"
Nov 03 12:07:10.179810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.179786    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" event={"ID":"1a4b8978-01ca-4005-99e5-fc3cbd4edb59","Type":"ContainerStarted","Data":"0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04"}
Nov 03 12:07:10.179908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.179823    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" event={"ID":"1a4b8978-01ca-4005-99e5-fc3cbd4edb59","Type":"ContainerStarted","Data":"c8ca24a4e136106c727f411d94a8ad5a400990ad1868db58e3be6934b5b7e380"}
Nov 03 12:07:10.187700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.187679    4546 scope.go:122] "RemoveContainer" containerID="d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652"
Nov 03 12:07:10.187979 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:07:10.187960    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652\": container with ID starting with d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652 not found: ID does not exist" containerID="d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652"
Nov 03 12:07:10.188039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.187985    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652"} err="failed to get container status \"d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652\": rpc error: code = NotFound desc = could not find container \"d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652\": container with ID starting with d5e712f0ca2a88404ab7813202cdb82f5e99dbf7e42422631f7257506e8e7652 not found: ID does not exist"
Nov 03 12:07:10.258809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.258779    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.258935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.258821    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.258935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.258861    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.258935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.258894    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.258935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.258923    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.259141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.258952    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.259141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259020    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.259141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259053    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.259141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259081    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk\") pod \"07785463-7365-4b0b-9ece-ebefa51c7469\" (UID: \"07785463-7365-4b0b-9ece-ebefa51c7469\") "
Nov 03 12:07:10.259332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259172    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:07:10.259332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:07:10.259431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259379    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:07:10.259431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259391    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:07:10.259802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259522    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:07:10.259931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259819    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.259931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259879    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07785463-7365-4b0b-9ece-ebefa51c7469-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.259931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259896    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.259931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.259910    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.261403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.261368    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:07:10.261403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.261383    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:07:10.261965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.261949    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "kube-api-access-86lrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:07:10.262053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.261956    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo" pod "07785463-7365-4b0b-9ece-ebefa51c7469" (UID: "07785463-7365-4b0b-9ece-ebefa51c7469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:07:10.360734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.360706    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.360734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.360732    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.360734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.360741    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07785463-7365-4b0b-9ece-ebefa51c7469-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.360734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.360762    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07785463-7365-4b0b-9ece-ebefa51c7469-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.360995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.360771    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lrk\" (UniqueName: \"kubernetes.io/projected/07785463-7365-4b0b-9ece-ebefa51c7469-kube-api-access-86lrk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:10.431030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.430951    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podStartSLOduration=2.430936425 podStartE2EDuration="2.430936425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:10.428358852 +0000 UTC m=+971.999942783" watchObservedRunningTime="2026-11-03 12:07:10.430936425 +0000 UTC m=+972.002520336"
Nov 03 12:07:10.592878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.592843    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"]
Nov 03 12:07:10.598982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.598956    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jb2gq"]
Nov 03 12:07:10.906122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:10.906090    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:12.573642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:12.573610    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" path="/var/lib/kubelet/pods/07785463-7365-4b0b-9ece-ebefa51c7469/volumes"
Nov 03 12:07:18.906458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:18.906427    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:18.919271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:18.919249    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:19.244724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:19.244691    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:07:25.426192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.426156    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"]
Nov 03 12:07:25.426623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.426610    4546 cpu_manager.go:414] "containerMap: removing container" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy"
Nov 03 12:07:25.426667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.426624    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy"
Nov 03 12:07:25.426743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.426734    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="07785463-7365-4b0b-9ece-ebefa51c7469" containerName="istio-proxy"
Nov 03 12:07:25.430388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.430371    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.482716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.482679    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"]
Nov 03 12:07:25.526003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.525957    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:07:25.526161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.526092    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:07:25.526161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.526151    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:07:25.526236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.526184    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:07:25.538743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.538716    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.538894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.538814    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.538894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.538858    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8ww\" (UniqueName: \"kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.640259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.640224    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.640424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.640328    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.640424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.640377    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8ww\" (UniqueName: \"kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.640900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.640876    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.642844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.642825    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.693183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.693126    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8ww\" (UniqueName: \"kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww\") pod \"maas-api-revoke-sub-b75e0f11-tpj58\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:25.743791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:25.743742    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:26.107591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:26.107567    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"]
Nov 03 12:07:26.108651 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:07:26.108628    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1db3853_7a87_4dde_8689_17239ce0d39d.slice/crio-08122639eede00517c3960f0962cb5e39f6692d523d90ff860d958e8baedd8ab WatchSource:0}: Error finding container 08122639eede00517c3960f0962cb5e39f6692d523d90ff860d958e8baedd8ab: Status 404 returned error can't find the container with id 08122639eede00517c3960f0962cb5e39f6692d523d90ff860d958e8baedd8ab
Nov 03 12:07:26.272249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:26.272217    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58" event={"ID":"c1db3853-7a87-4dde-8689-17239ce0d39d","Type":"ContainerStarted","Data":"08122639eede00517c3960f0962cb5e39f6692d523d90ff860d958e8baedd8ab"}
Nov 03 12:07:27.279903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:27.279827    4546 generic.go:356] "Generic (PLEG): container finished" podID="c1db3853-7a87-4dde-8689-17239ce0d39d" containerID="6f7d84a5e78c44f79f039ff130357e66d14058b403db2f307979cd746e2f7150" exitCode=0
Nov 03 12:07:27.280250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:27.279909    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58" event={"ID":"c1db3853-7a87-4dde-8689-17239ce0d39d","Type":"ContainerDied","Data":"6f7d84a5e78c44f79f039ff130357e66d14058b403db2f307979cd746e2f7150"}
Nov 03 12:07:28.431247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.431221    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:28.573834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.573730    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca\") pod \"c1db3853-7a87-4dde-8689-17239ce0d39d\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") "
Nov 03 12:07:28.574074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.574055    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww\" (UniqueName: \"kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww\") pod \"c1db3853-7a87-4dde-8689-17239ce0d39d\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") "
Nov 03 12:07:28.574176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.574105    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca\") pod \"c1db3853-7a87-4dde-8689-17239ce0d39d\" (UID: \"c1db3853-7a87-4dde-8689-17239ce0d39d\") "
Nov 03 12:07:28.574176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.574119    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca" pod "c1db3853-7a87-4dde-8689-17239ce0d39d" (UID: "c1db3853-7a87-4dde-8689-17239ce0d39d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:07:28.574555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.574533    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:28.576862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.576779    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww" pod "c1db3853-7a87-4dde-8689-17239ce0d39d" (UID: "c1db3853-7a87-4dde-8689-17239ce0d39d"). InnerVolumeSpecName "kube-api-access-9s8ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:07:28.576862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.576804    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca" pod "c1db3853-7a87-4dde-8689-17239ce0d39d" (UID: "c1db3853-7a87-4dde-8689-17239ce0d39d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:07:28.675669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.675644    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8ww\" (UniqueName: \"kubernetes.io/projected/c1db3853-7a87-4dde-8689-17239ce0d39d-kube-api-access-9s8ww\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:28.675669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:28.675668    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1db3853-7a87-4dde-8689-17239ce0d39d-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:29.292070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:29.292035    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58" event={"ID":"c1db3853-7a87-4dde-8689-17239ce0d39d","Type":"ContainerDied","Data":"08122639eede00517c3960f0962cb5e39f6692d523d90ff860d958e8baedd8ab"}
Nov 03 12:07:29.292247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:29.292114    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08122639eede00517c3960f0962cb5e39f6692d523d90ff860d958e8baedd8ab"
Nov 03 12:07:29.292247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:29.292048    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"
Nov 03 12:07:36.681478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.681445    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"]
Nov 03 12:07:36.681959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.681942    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c1db3853-7a87-4dde-8689-17239ce0d39d" containerName="revoke-keys"
Nov 03 12:07:36.682008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.681963    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1db3853-7a87-4dde-8689-17239ce0d39d" containerName="revoke-keys"
Nov 03 12:07:36.682071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.682061    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1db3853-7a87-4dde-8689-17239ce0d39d" containerName="revoke-keys"
Nov 03 12:07:36.685627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.685610    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.785430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.785401    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"]
Nov 03 12:07:36.861338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861306    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861342    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861466    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861508    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861589    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861626    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861659    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpxx\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861777    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.861833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.861817    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963087    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963128    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963203    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963257    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963305    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963366    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpxx\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963443    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963480    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963586    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963638    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963814    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.963901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.963880    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.964161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.964145    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.965710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.965677    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:36.965947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:36.965928    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:37.015987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.015959    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:37.016155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.016136    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpxx\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:37.299148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.299116    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:37.650870 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:07:37.650784    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:07:37.711375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.711349    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"]
Nov 03 12:07:37.712782 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:07:37.712714    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fec5de_3713_448e_a3aa_18c43d550178.slice/crio-818c747f59bceaa53778b7067adc4656afe33c440ab137be8ffaac3cd2086c21 WatchSource:0}: Error finding container 818c747f59bceaa53778b7067adc4656afe33c440ab137be8ffaac3cd2086c21: Status 404 returned error can't find the container with id 818c747f59bceaa53778b7067adc4656afe33c440ab137be8ffaac3cd2086c21
Nov 03 12:07:37.715297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.715267    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:07:37.715381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.715332    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:07:37.715381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:37.715362    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:07:38.349081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:38.349048    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" event={"ID":"57fec5de-3713-448e-a3aa-18c43d550178","Type":"ContainerStarted","Data":"57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c"}
Nov 03 12:07:38.349226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:38.349103    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" event={"ID":"57fec5de-3713-448e-a3aa-18c43d550178","Type":"ContainerStarted","Data":"818c747f59bceaa53778b7067adc4656afe33c440ab137be8ffaac3cd2086c21"}
Nov 03 12:07:38.506028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:38.505983    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podStartSLOduration=2.505970074 podStartE2EDuration="2.505970074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:38.503288094 +0000 UTC m=+1000.074872024" watchObservedRunningTime="2026-11-03 12:07:38.505970074 +0000 UTC m=+1000.077553985"
Nov 03 12:07:39.299762 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:39.299730    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:39.301596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:39.301570    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:39.301723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:39.301619    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 12:07:40.299879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.299847    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:40.300299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.299899    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 12:07:40.897538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.897506    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"]
Nov 03 12:07:40.902149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.902129    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:40.952740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.952715    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:07:40.955599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.955572    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"]
Nov 03 12:07:40.998839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.998809    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:07:40.998839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.998818    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:07:40.999012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:40.998849    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:07:41.032500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.032464    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jxw\" (UniqueName: \"kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.032693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.032594    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.032693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.032632    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.133545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.133513    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.133545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.133551    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.133811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.133657    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jxw\" (UniqueName: \"kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.134371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.134344    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.136420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.136393    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.186560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.186508    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jxw\" (UniqueName: \"kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw\") pod \"maas-api-revoke-sub-897a15ef-v5hg4\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.240953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.240925    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:41.309816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.300487    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:41.309816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.300558    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 12:07:41.643210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:41.643181    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"]
Nov 03 12:07:41.644838 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:07:41.644812    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0c509_562a_4a7e_b532_5ddbd5f8b941.slice/crio-59a06e0ed43d4c6b2a006d7aa6dac7b4bc5eb724b4521e376a39332db4cf82f8 WatchSource:0}: Error finding container 59a06e0ed43d4c6b2a006d7aa6dac7b4bc5eb724b4521e376a39332db4cf82f8: Status 404 returned error can't find the container with id 59a06e0ed43d4c6b2a006d7aa6dac7b4bc5eb724b4521e376a39332db4cf82f8
Nov 03 12:07:42.299860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:42.299783    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:42.299860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:42.299832    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 12:07:42.378396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:42.378363    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4" event={"ID":"e9d0c509-562a-4a7e-b532-5ddbd5f8b941","Type":"ContainerStarted","Data":"87900464c0de8306e5677c7be14f6c06928f08f2efecccd0c5d1b0d712021198"}
Nov 03 12:07:42.378813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:42.378413    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4" event={"ID":"e9d0c509-562a-4a7e-b532-5ddbd5f8b941","Type":"ContainerStarted","Data":"59a06e0ed43d4c6b2a006d7aa6dac7b4bc5eb724b4521e376a39332db4cf82f8"}
Nov 03 12:07:42.491989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:42.491940    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4" podStartSLOduration=2.491926456 podStartE2EDuration="2.491926456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:42.490158291 +0000 UTC m=+1004.061742200" watchObservedRunningTime="2026-11-03 12:07:42.491926456 +0000 UTC m=+1004.063510386"
Nov 03 12:07:43.299486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:43.299455    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:43.299668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:43.299504    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 12:07:43.385832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:43.385798    4546 generic.go:356] "Generic (PLEG): container finished" podID="e9d0c509-562a-4a7e-b532-5ddbd5f8b941" containerID="87900464c0de8306e5677c7be14f6c06928f08f2efecccd0c5d1b0d712021198" exitCode=0
Nov 03 12:07:43.386229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:43.385861    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4" event={"ID":"e9d0c509-562a-4a7e-b532-5ddbd5f8b941","Type":"ContainerDied","Data":"87900464c0de8306e5677c7be14f6c06928f08f2efecccd0c5d1b0d712021198"}
Nov 03 12:07:44.299534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.299500    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:44.299822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.299554    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 12:07:44.561780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.561688    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:44.599741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.599711    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca\") pod \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") "
Nov 03 12:07:44.599971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.599955    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw\" (UniqueName: \"kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw\") pod \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") "
Nov 03 12:07:44.600062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.600039    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca\") pod \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\" (UID: \"e9d0c509-562a-4a7e-b532-5ddbd5f8b941\") "
Nov 03 12:07:44.600441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.600414    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca" pod "e9d0c509-562a-4a7e-b532-5ddbd5f8b941" (UID: "e9d0c509-562a-4a7e-b532-5ddbd5f8b941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:07:44.601925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.601903    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca" pod "e9d0c509-562a-4a7e-b532-5ddbd5f8b941" (UID: "e9d0c509-562a-4a7e-b532-5ddbd5f8b941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:07:44.602099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.602078    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw" pod "e9d0c509-562a-4a7e-b532-5ddbd5f8b941" (UID: "e9d0c509-562a-4a7e-b532-5ddbd5f8b941"). InnerVolumeSpecName "kube-api-access-q2jxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:07:44.701787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.701733    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jxw\" (UniqueName: \"kubernetes.io/projected/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-kube-api-access-q2jxw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:44.701787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.701782    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:44.702018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:44.701795    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0c509-562a-4a7e-b532-5ddbd5f8b941-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:45.304584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:45.304556    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:45.304933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:45.304915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:45.305730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:45.305709    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:07:45.400343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:45.400310    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4" event={"ID":"e9d0c509-562a-4a7e-b532-5ddbd5f8b941","Type":"ContainerDied","Data":"59a06e0ed43d4c6b2a006d7aa6dac7b4bc5eb724b4521e376a39332db4cf82f8"}
Nov 03 12:07:45.400343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:45.400332    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"
Nov 03 12:07:45.400561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:45.400358    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59a06e0ed43d4c6b2a006d7aa6dac7b4bc5eb724b4521e376a39332db4cf82f8"
Nov 03 12:07:50.794589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.794558    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"]
Nov 03 12:07:50.795248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.795228    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e9d0c509-562a-4a7e-b532-5ddbd5f8b941" containerName="revoke-keys"
Nov 03 12:07:50.795345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.795250    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d0c509-562a-4a7e-b532-5ddbd5f8b941" containerName="revoke-keys"
Nov 03 12:07:50.795407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.795384    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d0c509-562a-4a7e-b532-5ddbd5f8b941" containerName="revoke-keys"
Nov 03 12:07:50.798964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.798943    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.865199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.865174    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822g8\" (UniqueName: \"kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.865315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.865212    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.865395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.865336    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.895280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.895253    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:07:50.895280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.895254    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:07:50.895466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.895292    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:07:50.895466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.895385    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:07:50.898151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.898126    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"]
Nov 03 12:07:50.966425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.966399    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.966550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.966534    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822g8\" (UniqueName: \"kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.966591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.966575    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.967198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.967169    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:50.969036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:50.969019    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:51.020182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:51.020161    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822g8\" (UniqueName: \"kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8\") pod \"maas-api-revoke-sub-99eff651-55cv8\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:51.111855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:51.111835    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:51.475951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:51.475921    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"]
Nov 03 12:07:51.477501 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:07:51.477475    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677916aa_2437_4d24_99f2_7957c96918e2.slice/crio-91649ec0548c7a35207b23e217183efe7ca7783bf488c5bf2aad721da5e773cf WatchSource:0}: Error finding container 91649ec0548c7a35207b23e217183efe7ca7783bf488c5bf2aad721da5e773cf: Status 404 returned error can't find the container with id 91649ec0548c7a35207b23e217183efe7ca7783bf488c5bf2aad721da5e773cf
Nov 03 12:07:52.440567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:52.440530    4546 generic.go:356] "Generic (PLEG): container finished" podID="677916aa-2437-4d24-99f2-7957c96918e2" containerID="6b4d13408b62cf5a9c7842f433b44c5db458ab9c74c485211ac04f113d56272f" exitCode=0
Nov 03 12:07:52.440946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:52.440592    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8" event={"ID":"677916aa-2437-4d24-99f2-7957c96918e2","Type":"ContainerDied","Data":"6b4d13408b62cf5a9c7842f433b44c5db458ab9c74c485211ac04f113d56272f"}
Nov 03 12:07:52.440946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:52.440656    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8" event={"ID":"677916aa-2437-4d24-99f2-7957c96918e2","Type":"ContainerStarted","Data":"91649ec0548c7a35207b23e217183efe7ca7783bf488c5bf2aad721da5e773cf"}
Nov 03 12:07:53.668676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.668645    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:53.820786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.799105    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8\" (UniqueName: \"kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8\") pod \"677916aa-2437-4d24-99f2-7957c96918e2\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") "
Nov 03 12:07:53.820786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.799225    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca\") pod \"677916aa-2437-4d24-99f2-7957c96918e2\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") "
Nov 03 12:07:53.820786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.799290    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca\") pod \"677916aa-2437-4d24-99f2-7957c96918e2\" (UID: \"677916aa-2437-4d24-99f2-7957c96918e2\") "
Nov 03 12:07:53.850979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.837952    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca" pod "677916aa-2437-4d24-99f2-7957c96918e2" (UID: "677916aa-2437-4d24-99f2-7957c96918e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:07:53.871812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.865961    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca" pod "677916aa-2437-4d24-99f2-7957c96918e2" (UID: "677916aa-2437-4d24-99f2-7957c96918e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:07:53.925068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.925014    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8" pod "677916aa-2437-4d24-99f2-7957c96918e2" (UID: "677916aa-2437-4d24-99f2-7957c96918e2"). InnerVolumeSpecName "kube-api-access-822g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:07:53.980393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.980314    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:53.980393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.980352    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822g8\" (UniqueName: \"kubernetes.io/projected/677916aa-2437-4d24-99f2-7957c96918e2-kube-api-access-822g8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:53.980393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:53.980370    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677916aa-2437-4d24-99f2-7957c96918e2-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:54.466603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:54.466566    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8" event={"ID":"677916aa-2437-4d24-99f2-7957c96918e2","Type":"ContainerDied","Data":"91649ec0548c7a35207b23e217183efe7ca7783bf488c5bf2aad721da5e773cf"}
Nov 03 12:07:54.470935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:54.470914    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91649ec0548c7a35207b23e217183efe7ca7783bf488c5bf2aad721da5e773cf"
Nov 03 12:07:54.471163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:54.471149    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"
Nov 03 12:07:56.444294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.444264    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"]
Nov 03 12:07:56.444721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.444708    4546 cpu_manager.go:414] "containerMap: removing container" podUID="677916aa-2437-4d24-99f2-7957c96918e2" containerName="revoke-keys"
Nov 03 12:07:56.444788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.444722    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677916aa-2437-4d24-99f2-7957c96918e2" containerName="revoke-keys"
Nov 03 12:07:56.444851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.444834    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="677916aa-2437-4d24-99f2-7957c96918e2" containerName="revoke-keys"
Nov 03 12:07:56.448243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.448226    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.501624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.501573    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"]
Nov 03 12:07:56.541674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.541637    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.541856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.541802    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr62r\" (UniqueName: \"kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.541856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.541851    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.545081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.545056    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:07:56.545081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.545070    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:07:56.545081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.545059    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:07:56.545320 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.545115    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:07:56.642464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.642430    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.642616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.642497    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr62r\" (UniqueName: \"kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.642616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.642532    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.643100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.643070    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.644889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.644851    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.694888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.694834    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr62r\" (UniqueName: \"kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r\") pod \"maas-api-revoke-sub-e2c92f0d-gcjd8\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:56.766946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:56.766921    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:57.139976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:57.139951    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"]
Nov 03 12:07:57.141611 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:07:57.141585    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1b67c5_d3cf_47fa_8730_747df05791db.slice/crio-7bf2cf57b6adab7dddf5dcf585f874180fc2c610248b95afa427be072077b188 WatchSource:0}: Error finding container 7bf2cf57b6adab7dddf5dcf585f874180fc2c610248b95afa427be072077b188: Status 404 returned error can't find the container with id 7bf2cf57b6adab7dddf5dcf585f874180fc2c610248b95afa427be072077b188
Nov 03 12:07:57.484118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:57.484084    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8" event={"ID":"fc1b67c5-d3cf-47fa-8730-747df05791db","Type":"ContainerStarted","Data":"d2ebbd0e347b130424ac101c8857d371c19740043158f3a53e992bc651ee3084"}
Nov 03 12:07:57.484414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:57.484130    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8" event={"ID":"fc1b67c5-d3cf-47fa-8730-747df05791db","Type":"ContainerStarted","Data":"7bf2cf57b6adab7dddf5dcf585f874180fc2c610248b95afa427be072077b188"}
Nov 03 12:07:57.592031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:57.591983    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8" podStartSLOduration=1.591969688 podStartE2EDuration="1.591969688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:57.58850184 +0000 UTC m=+1019.160085763" watchObservedRunningTime="2026-11-03 12:07:57.591969688 +0000 UTC m=+1019.163553598"
Nov 03 12:07:58.491420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:58.491390    4546 generic.go:356] "Generic (PLEG): container finished" podID="fc1b67c5-d3cf-47fa-8730-747df05791db" containerID="d2ebbd0e347b130424ac101c8857d371c19740043158f3a53e992bc651ee3084" exitCode=0
Nov 03 12:07:58.491811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:58.491480    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8" event={"ID":"fc1b67c5-d3cf-47fa-8730-747df05791db","Type":"ContainerDied","Data":"d2ebbd0e347b130424ac101c8857d371c19740043158f3a53e992bc651ee3084"}
Nov 03 12:07:59.640715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.640689    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:07:59.674783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.674741    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca\") pod \"fc1b67c5-d3cf-47fa-8730-747df05791db\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") "
Nov 03 12:07:59.674889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.674836    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r\" (UniqueName: \"kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r\") pod \"fc1b67c5-d3cf-47fa-8730-747df05791db\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") "
Nov 03 12:07:59.674970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.674953    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca\") pod \"fc1b67c5-d3cf-47fa-8730-747df05791db\" (UID: \"fc1b67c5-d3cf-47fa-8730-747df05791db\") "
Nov 03 12:07:59.675358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.675280    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca" pod "fc1b67c5-d3cf-47fa-8730-747df05791db" (UID: "fc1b67c5-d3cf-47fa-8730-747df05791db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:07:59.675547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.675421    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:59.676870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.676844    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca" pod "fc1b67c5-d3cf-47fa-8730-747df05791db" (UID: "fc1b67c5-d3cf-47fa-8730-747df05791db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:07:59.677065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.677042    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r" pod "fc1b67c5-d3cf-47fa-8730-747df05791db" (UID: "fc1b67c5-d3cf-47fa-8730-747df05791db"). InnerVolumeSpecName "kube-api-access-dr62r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:07:59.777053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.776976    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr62r\" (UniqueName: \"kubernetes.io/projected/fc1b67c5-d3cf-47fa-8730-747df05791db-kube-api-access-dr62r\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:07:59.777053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:07:59.777004    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1b67c5-d3cf-47fa-8730-747df05791db-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:00.504045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:00.504009    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8" event={"ID":"fc1b67c5-d3cf-47fa-8730-747df05791db","Type":"ContainerDied","Data":"7bf2cf57b6adab7dddf5dcf585f874180fc2c610248b95afa427be072077b188"}
Nov 03 12:08:00.504045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:00.504031    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"
Nov 03 12:08:00.504244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:00.504092    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf2cf57b6adab7dddf5dcf585f874180fc2c610248b95afa427be072077b188"
Nov 03 12:08:01.559369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.559339    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"]
Nov 03 12:08:01.559891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.559875    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fc1b67c5-d3cf-47fa-8730-747df05791db" containerName="revoke-keys"
Nov 03 12:08:01.559891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.559891    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1b67c5-d3cf-47fa-8730-747df05791db" containerName="revoke-keys"
Nov 03 12:08:01.560004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.559984    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1b67c5-d3cf-47fa-8730-747df05791db" containerName="revoke-keys"
Nov 03 12:08:01.563366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.563346    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.598438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.598410    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.598592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.598473    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxlk\" (UniqueName: \"kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.598592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.598571    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.611872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.611839    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:08:01.657818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.657786    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:08:01.657950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.657821    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:08:01.657950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.657866    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:08:01.661047 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.661026    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"]
Nov 03 12:08:01.699879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.699854    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.700023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.699935    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxlk\" (UniqueName: \"kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.700023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.699975    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.700459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.700434    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.702455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.702431    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.752657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.752631    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxlk\" (UniqueName: \"kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk\") pod \"maas-api-revoke-sub-d31e0199-v2kf8\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:01.877333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:01.877258    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:02.243503 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:02.243479    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"]
Nov 03 12:08:02.245218 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:08:02.245193    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c4ea78_ec92_4dcd_bcae_458d684d6716.slice/crio-548d7be174bba5e673ffe97222c219fad39ad7fbc8ae3cffc367338d9a6027bd WatchSource:0}: Error finding container 548d7be174bba5e673ffe97222c219fad39ad7fbc8ae3cffc367338d9a6027bd: Status 404 returned error can't find the container with id 548d7be174bba5e673ffe97222c219fad39ad7fbc8ae3cffc367338d9a6027bd
Nov 03 12:08:02.516824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:02.516797    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8" event={"ID":"41c4ea78-ec92-4dcd-bcae-458d684d6716","Type":"ContainerStarted","Data":"548d7be174bba5e673ffe97222c219fad39ad7fbc8ae3cffc367338d9a6027bd"}
Nov 03 12:08:03.524544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:03.524503    4546 generic.go:356] "Generic (PLEG): container finished" podID="41c4ea78-ec92-4dcd-bcae-458d684d6716" containerID="2f6e6add1d41473a801d9daa9e1478467c9ba6b6ab324d0d30bb6d04de3b2189" exitCode=0
Nov 03 12:08:03.525080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:03.524580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8" event={"ID":"41c4ea78-ec92-4dcd-bcae-458d684d6716","Type":"ContainerDied","Data":"2f6e6add1d41473a801d9daa9e1478467c9ba6b6ab324d0d30bb6d04de3b2189"}
Nov 03 12:08:04.675997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.675974    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:04.733287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.733251    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca\") pod \"41c4ea78-ec92-4dcd-bcae-458d684d6716\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") "
Nov 03 12:08:04.733421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.733335    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca\") pod \"41c4ea78-ec92-4dcd-bcae-458d684d6716\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") "
Nov 03 12:08:04.733421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.733364    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk\" (UniqueName: \"kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk\") pod \"41c4ea78-ec92-4dcd-bcae-458d684d6716\" (UID: \"41c4ea78-ec92-4dcd-bcae-458d684d6716\") "
Nov 03 12:08:04.733683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.733649    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca" pod "41c4ea78-ec92-4dcd-bcae-458d684d6716" (UID: "41c4ea78-ec92-4dcd-bcae-458d684d6716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:08:04.735499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.735477    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca" pod "41c4ea78-ec92-4dcd-bcae-458d684d6716" (UID: "41c4ea78-ec92-4dcd-bcae-458d684d6716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:08:04.735595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.735521    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk" pod "41c4ea78-ec92-4dcd-bcae-458d684d6716" (UID: "41c4ea78-ec92-4dcd-bcae-458d684d6716"). InnerVolumeSpecName "kube-api-access-ccxlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:08:04.834870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.834792    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:04.834870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.834819    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c4ea78-ec92-4dcd-bcae-458d684d6716-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:04.834870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:04.834829    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxlk\" (UniqueName: \"kubernetes.io/projected/41c4ea78-ec92-4dcd-bcae-458d684d6716-kube-api-access-ccxlk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:05.537937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:05.537902    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"
Nov 03 12:08:05.538118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:05.537937    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8" event={"ID":"41c4ea78-ec92-4dcd-bcae-458d684d6716","Type":"ContainerDied","Data":"548d7be174bba5e673ffe97222c219fad39ad7fbc8ae3cffc367338d9a6027bd"}
Nov 03 12:08:05.538118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:05.537985    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="548d7be174bba5e673ffe97222c219fad39ad7fbc8ae3cffc367338d9a6027bd"
Nov 03 12:08:07.649954 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:08:07.649870    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:08:16.428414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.428378    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"]
Nov 03 12:08:16.428967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.428932    4546 cpu_manager.go:414] "containerMap: removing container" podUID="41c4ea78-ec92-4dcd-bcae-458d684d6716" containerName="revoke-keys"
Nov 03 12:08:16.428967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.428947    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c4ea78-ec92-4dcd-bcae-458d684d6716" containerName="revoke-keys"
Nov 03 12:08:16.429049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.429021    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c4ea78-ec92-4dcd-bcae-458d684d6716" containerName="revoke-keys"
Nov 03 12:08:16.432463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.432441    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.485979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.485949    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"]
Nov 03 12:08:16.528930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.528899    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7ab4514d17b9\""
Nov 03 12:08:16.529094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.528988    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:08:16.529094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.529044    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:08:16.529094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.529052    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4e5279dc91ae\""
Nov 03 12:08:16.529094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.529068    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:08:16.529094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.529089    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:08:16.529286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.529116    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-x4r42\""
Nov 03 12:08:16.584668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.584638    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.584882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.584801    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.584934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.584910    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdtr\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.584969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.584939    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.685908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.685826    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdtr\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.685908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.685876    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.686125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.685917    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.686125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.686025    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.686731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.686670    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.686731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.686698    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.688447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.688427    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.717178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.717148    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"]
Nov 03 12:08:16.721245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.721225    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.743562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.743520    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdtr\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr\") pod \"payload-processing-external-model-65665d6cd9-76q84\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") " pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.745858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.745710    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:16.771337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.771314    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5951bc-openshift-default-dockercfg-bp9bz\""
Nov 03 12:08:16.819638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.819607    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"]
Nov 03 12:08:16.890268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890230    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890269    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890301    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890470    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890518    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890576    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfmh\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890730    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890850    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.890937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.890919    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991478    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991552    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991592    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991633    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991658    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991701    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991765    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991799    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.991938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.991847    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfmh\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.992150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.992041    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.992150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.992109    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.992417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.992288    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.992485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.992464    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.992620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.992601    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.993994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.993971    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:16.994761 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:16.994726    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:17.044406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.044369    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:17.044540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.044471    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfmh\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:17.148065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.148034    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"]
Nov 03 12:08:17.149379 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:08:17.149357    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17326d8_7c07_4e33_b805_6287920734e5.slice/crio-1d905aca205c357179bc54f4f4cb06c36bb553ba5f0d1cd7f398db4de150ebfe WatchSource:0}: Error finding container 1d905aca205c357179bc54f4f4cb06c36bb553ba5f0d1cd7f398db4de150ebfe: Status 404 returned error can't find the container with id 1d905aca205c357179bc54f4f4cb06c36bb553ba5f0d1cd7f398db4de150ebfe
Nov 03 12:08:17.344073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.343992    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:17.611661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.611585    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84" event={"ID":"e17326d8-7c07-4e33-b805-6287920734e5","Type":"ContainerStarted","Data":"012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116"}
Nov 03 12:08:17.611661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.611636    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84" event={"ID":"e17326d8-7c07-4e33-b805-6287920734e5","Type":"ContainerStarted","Data":"1d905aca205c357179bc54f4f4cb06c36bb553ba5f0d1cd7f398db4de150ebfe"}
Nov 03 12:08:17.612100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.611800    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:17.722523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.722480    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84" podStartSLOduration=1.722466817 podStartE2EDuration="1.722466817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:17.719635198 +0000 UTC m=+1039.291219120" watchObservedRunningTime="2026-11-03 12:08:17.722466817 +0000 UTC m=+1039.294050728"
Nov 03 12:08:17.779977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.779952    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"]
Nov 03 12:08:17.780860 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:08:17.780829    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5890d46_715d_45cb_8121_d744306340eb.slice/crio-b3684089ed31c5cc1fab41eef0bd88b4015e886c184a1ef08cdea60ae917d474 WatchSource:0}: Error finding container b3684089ed31c5cc1fab41eef0bd88b4015e886c184a1ef08cdea60ae917d474: Status 404 returned error can't find the container with id b3684089ed31c5cc1fab41eef0bd88b4015e886c184a1ef08cdea60ae917d474
Nov 03 12:08:17.783561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.783531    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:17.783649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.783606    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:17.783718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:17.783653    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:18.637022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:18.636975    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" event={"ID":"d5890d46-715d-45cb-8121-d744306340eb","Type":"ContainerStarted","Data":"24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177"}
Nov 03 12:08:18.637022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:18.637030    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" event={"ID":"d5890d46-715d-45cb-8121-d744306340eb","Type":"ContainerStarted","Data":"b3684089ed31c5cc1fab41eef0bd88b4015e886c184a1ef08cdea60ae917d474"}
Nov 03 12:08:18.838017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:18.837965    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podStartSLOduration=2.837949349 podStartE2EDuration="2.837949349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:18.790998012 +0000 UTC m=+1040.362581942" watchObservedRunningTime="2026-11-03 12:08:18.837949349 +0000 UTC m=+1040.409533261"
Nov 03 12:08:19.345093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:19.345066    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:19.346920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:19.346893    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:08:19.347070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:19.346943    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 12:08:20.344829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:20.344786    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:08:20.345266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:20.344870    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 12:08:21.345401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:21.345366    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:08:21.345841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:21.345429    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 12:08:22.344839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:22.344801    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:08:22.345031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:22.344861    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 12:08:23.345031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:23.344997    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:08:23.345427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:23.345059    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 12:08:24.089714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.089680    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"]
Nov 03 12:08:24.094005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.093979    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.189860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.189829    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:08:24.190208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.190083    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:08:24.190208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.190126    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:08:24.190208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.190135    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:08:24.194474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.194442    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"]
Nov 03 12:08:24.196918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.196894    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkm9m\" (UniqueName: \"kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.197068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.197051    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.197221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.197201    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.298305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.298267    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.298496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.298422    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.298496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.298452    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkm9m\" (UniqueName: \"kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.299110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.299081    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.301139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.301116    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.351894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.351813    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkm9m\" (UniqueName: \"kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m\") pod \"maas-api-revoke-sub-196661de-7825s\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.407898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.407864    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:24.501229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.501196    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:24.501975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.501943    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:24.502978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.502960    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 12:08:24.778486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:24.778441    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"]
Nov 03 12:08:24.778641 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:08:24.778617    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135b1d9c_b2fa_4838_a176_ff7c840c9080.slice/crio-b93d84099fedc306668980fe82231c29b2496cedec2629472d50ef3b3e4ceec7 WatchSource:0}: Error finding container b93d84099fedc306668980fe82231c29b2496cedec2629472d50ef3b3e4ceec7: Status 404 returned error can't find the container with id b93d84099fedc306668980fe82231c29b2496cedec2629472d50ef3b3e4ceec7
Nov 03 12:08:25.684772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:25.684444    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s" event={"ID":"135b1d9c-b2fa-4838-a176-ff7c840c9080","Type":"ContainerStarted","Data":"644489d74c6bda42c8f13d7ccafe1739631e285314df5a4284c6eb39afd87546"}
Nov 03 12:08:25.684772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:25.684543    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s" event={"ID":"135b1d9c-b2fa-4838-a176-ff7c840c9080","Type":"ContainerStarted","Data":"b93d84099fedc306668980fe82231c29b2496cedec2629472d50ef3b3e4ceec7"}
Nov 03 12:08:25.793443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:25.793381    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s" podStartSLOduration=1.7933615710000002 podStartE2EDuration="1.793361571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:25.790079608 +0000 UTC m=+1047.361663560" watchObservedRunningTime="2026-11-03 12:08:25.793361571 +0000 UTC m=+1047.364945483"
Nov 03 12:08:26.691411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:26.691374    4546 generic.go:356] "Generic (PLEG): container finished" podID="135b1d9c-b2fa-4838-a176-ff7c840c9080" containerID="644489d74c6bda42c8f13d7ccafe1739631e285314df5a4284c6eb39afd87546" exitCode=0
Nov 03 12:08:26.691857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:26.691459    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s" event={"ID":"135b1d9c-b2fa-4838-a176-ff7c840c9080","Type":"ContainerDied","Data":"644489d74c6bda42c8f13d7ccafe1739631e285314df5a4284c6eb39afd87546"}
Nov 03 12:08:27.841785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.841761    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:27.957359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.957284    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m\" (UniqueName: \"kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m\") pod \"135b1d9c-b2fa-4838-a176-ff7c840c9080\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") "
Nov 03 12:08:27.957359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.957324    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca\") pod \"135b1d9c-b2fa-4838-a176-ff7c840c9080\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") "
Nov 03 12:08:27.957566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.957362    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca\") pod \"135b1d9c-b2fa-4838-a176-ff7c840c9080\" (UID: \"135b1d9c-b2fa-4838-a176-ff7c840c9080\") "
Nov 03 12:08:27.957821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.957786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca" pod "135b1d9c-b2fa-4838-a176-ff7c840c9080" (UID: "135b1d9c-b2fa-4838-a176-ff7c840c9080"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:08:27.958133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.957999    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:27.959684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.959663    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca" pod "135b1d9c-b2fa-4838-a176-ff7c840c9080" (UID: "135b1d9c-b2fa-4838-a176-ff7c840c9080"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:08:27.959792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:27.959713    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m" pod "135b1d9c-b2fa-4838-a176-ff7c840c9080" (UID: "135b1d9c-b2fa-4838-a176-ff7c840c9080"). InnerVolumeSpecName "kube-api-access-mkm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:08:28.059147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:28.059112    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkm9m\" (UniqueName: \"kubernetes.io/projected/135b1d9c-b2fa-4838-a176-ff7c840c9080-kube-api-access-mkm9m\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:28.059147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:28.059141    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135b1d9c-b2fa-4838-a176-ff7c840c9080-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:08:28.640494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:28.640465    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:08:28.704703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:28.704678    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"
Nov 03 12:08:28.704889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:28.704679    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s" event={"ID":"135b1d9c-b2fa-4838-a176-ff7c840c9080","Type":"ContainerDied","Data":"b93d84099fedc306668980fe82231c29b2496cedec2629472d50ef3b3e4ceec7"}
Nov 03 12:08:28.704889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:28.704794    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93d84099fedc306668980fe82231c29b2496cedec2629472d50ef3b3e4ceec7"
Nov 03 12:08:35.690123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.690093    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"]
Nov 03 12:08:35.690583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.690571    4546 cpu_manager.go:414] "containerMap: removing container" podUID="135b1d9c-b2fa-4838-a176-ff7c840c9080" containerName="revoke-keys"
Nov 03 12:08:35.690627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.690585    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135b1d9c-b2fa-4838-a176-ff7c840c9080" containerName="revoke-keys"
Nov 03 12:08:35.690675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.690667    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="135b1d9c-b2fa-4838-a176-ff7c840c9080" containerName="revoke-keys"
Nov 03 12:08:35.694294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.694274    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.736611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.736582    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"]
Nov 03 12:08:35.740877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.740857    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.783503 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.783445    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"]
Nov 03 12:08:35.826391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.826355    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"]
Nov 03 12:08:35.847856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.847823    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.847979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.847875    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.847979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.847909    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.847978    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848059    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848107    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848127    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848227    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848306    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlgj\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848356    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848387    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848447    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848503    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848567    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848625    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848671    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.848778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848740    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnwz\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.848826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.848801    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.949834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949710    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.949834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949789    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.949834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949819    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949842    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949869    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949923    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949953    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.949981    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950024    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlgj\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950074    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950105    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950131    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950172    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950238    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950239    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950267    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950315    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950348    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnwz\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950386    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.950565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950538    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.951008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950817    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.951008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.950984    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.951123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.951076    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.951261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.951234    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.951372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.951350    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.951514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.951491    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.951694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.951666    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.952159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.952092    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.953070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.953039    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.953807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.953675    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:35.953941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.953918    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:35.954012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:35.953981    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:36.003530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.003502    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:36.003643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.003621    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:36.003852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.003835    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlgj\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5cvqk\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:36.003902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.003874    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnwz\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dhc85\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:36.008725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.008704    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:36.055205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.055172    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:36.423202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.423168    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"]
Nov 03 12:08:36.460440 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:08:36.460413    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea624674_bd74_4592_9a75_cc59fcaa619f.slice/crio-c53e2c75e1f5c7f05e2fd07cdae8392e5bb05fdd58f6650febd00c299a0111ad WatchSource:0}: Error finding container c53e2c75e1f5c7f05e2fd07cdae8392e5bb05fdd58f6650febd00c299a0111ad: Status 404 returned error can't find the container with id c53e2c75e1f5c7f05e2fd07cdae8392e5bb05fdd58f6650febd00c299a0111ad
Nov 03 12:08:36.462625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.462595    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:36.462693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.462662    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:36.462693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.462690    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:36.471438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.471420    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"]
Nov 03 12:08:36.473213 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:08:36.473186    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9391c99_e893_4007_9dac_6ac0f722953f.slice/crio-f754a8477ae668f19b894487f26bdf37ad6060ab9b809ed5c235f07171ca9d59 WatchSource:0}: Error finding container f754a8477ae668f19b894487f26bdf37ad6060ab9b809ed5c235f07171ca9d59: Status 404 returned error can't find the container with id f754a8477ae668f19b894487f26bdf37ad6060ab9b809ed5c235f07171ca9d59
Nov 03 12:08:36.475636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.475608    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:36.475700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.475672    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:36.475740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.475701    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:08:36.754587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.753880    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" event={"ID":"a9391c99-e893-4007-9dac-6ac0f722953f","Type":"ContainerStarted","Data":"4d8d2786750ce14071cf05ded403bc7d81863e4f2a82cc4fb8a128527c06a1a0"}
Nov 03 12:08:36.754587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.753944    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" event={"ID":"a9391c99-e893-4007-9dac-6ac0f722953f","Type":"ContainerStarted","Data":"f754a8477ae668f19b894487f26bdf37ad6060ab9b809ed5c235f07171ca9d59"}
Nov 03 12:08:36.755996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.755961    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" event={"ID":"ea624674-bd74-4592-9a75-cc59fcaa619f","Type":"ContainerStarted","Data":"c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75"}
Nov 03 12:08:36.756115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.756017    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" event={"ID":"ea624674-bd74-4592-9a75-cc59fcaa619f","Type":"ContainerStarted","Data":"c53e2c75e1f5c7f05e2fd07cdae8392e5bb05fdd58f6650febd00c299a0111ad"}
Nov 03 12:08:36.864406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.864342    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podStartSLOduration=1.8643228170000001 podStartE2EDuration="1.864322817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:36.860999541 +0000 UTC m=+1058.432583453" watchObservedRunningTime="2026-11-03 12:08:36.864322817 +0000 UTC m=+1058.435906730"
Nov 03 12:08:36.969939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:36.969884    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podStartSLOduration=1.9698699880000001 podStartE2EDuration="1.969869988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:36.967190687 +0000 UTC m=+1058.538774647" watchObservedRunningTime="2026-11-03 12:08:36.969869988 +0000 UTC m=+1058.541453959"
Nov 03 12:08:37.019835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:37.013178    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:37.020015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:37.019885    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:37.020015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:37.019959    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:37.058039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:37.057809    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:37.058039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:37.057923    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:37.058039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:37.058021    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:37.682107 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:08:37.682065    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:08:38.030054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:38.029568    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:38.030054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:38.029630    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:38.057276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:38.057242    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:38.057554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:38.057525    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:39.011268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:39.011229    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:39.038943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:39.038900    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:39.058793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:39.058401    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:39.058793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:39.058461    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:40.011787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:40.011108    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:40.011787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:40.011183    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:40.058229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:40.056977    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:40.058229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:40.057053    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:41.023775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:41.020077    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:41.023775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:41.020134    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:41.062880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:41.062845    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:41.063494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:41.063423    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:42.016775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:42.011117    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:42.016775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:42.011189    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:42.057775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:42.057318    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:42.057775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:42.057364    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:43.009155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:43.009122    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:43.009707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:43.009180    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:43.055809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:43.055773    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:43.055970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:43.055837    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:44.009436 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:44.009340    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 12:08:44.009436 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:44.009410    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 12:08:44.056489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:44.056446    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:08:44.056706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:44.056515    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 12:08:45.984489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:45.984444    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:45.985254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:45.985177    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:45.986358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:45.986339    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:08:46.052922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:46.052886    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:08:46.053212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:46.053184    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:08:46.139142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:46.139110    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:46.140465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:46.140446    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:08:46.142117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:08:46.142095    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 12:09:05.100806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.100769    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"]
Nov 03 12:09:05.106170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.106145    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.207668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.207616    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"]
Nov 03 12:09:05.227407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227368    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227471    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227503    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227534    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227616    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227665    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227685    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227718    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcnnr\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.227983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.227958    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329243    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329281    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329311    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329340    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329370    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329389    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329422    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcnnr\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329527    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329619    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329795    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.329927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.329886    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.330073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.330018    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.330073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.330032    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.330670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.330644    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.331984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.331963    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.332218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.332199    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.382552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.382484    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.382805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.382782    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcnnr\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.419528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.419484    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:05.839861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.839830    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"]
Nov 03 12:09:05.840568 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:09:05.840538    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611e4d34_68ad_454c_8088_5b42b559d36e.slice/crio-c506d36b7f75e10d15bfd5f6183aafa66b188babdcbac566763f35982228b062 WatchSource:0}: Error finding container c506d36b7f75e10d15bfd5f6183aafa66b188babdcbac566763f35982228b062: Status 404 returned error can't find the container with id c506d36b7f75e10d15bfd5f6183aafa66b188babdcbac566763f35982228b062
Nov 03 12:09:05.842784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.842737    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:09:05.842859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.842832    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:09:05.842895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:05.842868    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:09:06.005164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:06.005128    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" event={"ID":"611e4d34-68ad-454c-8088-5b42b559d36e","Type":"ContainerStarted","Data":"3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304"}
Nov 03 12:09:06.005366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:06.005196    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" event={"ID":"611e4d34-68ad-454c-8088-5b42b559d36e","Type":"ContainerStarted","Data":"c506d36b7f75e10d15bfd5f6183aafa66b188babdcbac566763f35982228b062"}
Nov 03 12:09:06.130367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:06.130241    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podStartSLOduration=1.130221303 podStartE2EDuration="1.130221303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:06.128448317 +0000 UTC m=+1087.700032242" watchObservedRunningTime="2026-11-03 12:09:06.130221303 +0000 UTC m=+1087.701805214"
Nov 03 12:09:06.423143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:06.423035    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:06.425391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:06.425355    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:06.425687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:06.425657    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:07.432262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:07.432224    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:07.432826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:07.432293    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:07.659177 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:07.659134    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:09:08.432239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:08.432195    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:08.432430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:08.432255    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:09.425772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:09.425115    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:09.425772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:09.425174    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:10.425186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:10.425148    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:10.425795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:10.425763    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:11.428774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:11.428144    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:11.428774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:11.428205    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:12.420883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:12.420845    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:09:12.421057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:12.420900    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 12:09:13.425970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:13.425926    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:13.426486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:13.426365    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:13.427303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:13.427273    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:09:28.405777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.405482    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"]
Nov 03 12:09:28.419887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.419855    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.556778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.533593    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.557129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.557096    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.564777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.557414    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2r5\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.578774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.568912    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.649844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.649804    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"]
Nov 03 12:09:28.695167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.695078    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.695398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.695376    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.695628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.695609    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2r5\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.695779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.695737    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.697324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.697276    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.698243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.698220    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.763294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.763214    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:28.767320 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:28.767251    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2r5\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5\") pod \"payload-processing-external-model-778d4c669c-h9jrl\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:29.069035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:29.068326    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:29.464220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:29.464195    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"]
Nov 03 12:09:29.465419 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:09:29.465385    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d58e22e_2b44_4736_bdfd_8d5924b04317.slice/crio-7f9ebbcc0ee59c09e8b55711e1482c471cd7d3a7647fe1fb4c4707e9c6a6e19d WatchSource:0}: Error finding container 7f9ebbcc0ee59c09e8b55711e1482c471cd7d3a7647fe1fb4c4707e9c6a6e19d: Status 404 returned error can't find the container with id 7f9ebbcc0ee59c09e8b55711e1482c471cd7d3a7647fe1fb4c4707e9c6a6e19d
Nov 03 12:09:30.185797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:30.185759    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" event={"ID":"6d58e22e-2b44-4736-bdfd-8d5924b04317","Type":"ContainerStarted","Data":"81f7f6f499c328e8a46cc6ef1ac11a3c5796816a04509d4b48ea7319b6ad2b1b"}
Nov 03 12:09:30.186008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:30.185809    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" event={"ID":"6d58e22e-2b44-4736-bdfd-8d5924b04317","Type":"ContainerStarted","Data":"7f9ebbcc0ee59c09e8b55711e1482c471cd7d3a7647fe1fb4c4707e9c6a6e19d"}
Nov 03 12:09:30.186008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:30.185881    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:30.293305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:30.293247    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" podStartSLOduration=2.293232512 podStartE2EDuration="2.293232512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:30.291815958 +0000 UTC m=+1111.863399889" watchObservedRunningTime="2026-11-03 12:09:30.293232512 +0000 UTC m=+1111.864816424"
Nov 03 12:09:35.680826 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:35.680625    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4e5279dc91ae: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:35.680826 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:35.680660    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65665d6cd9-76q84: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:35.680826 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:35.680740    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials podName:e17326d8-7c07-4e33-b805-6287920734e5 nodeName:}" failed. No retries permitted until 2026-11-03 12:09:36.180720001 +0000 UTC m=+1117.752303905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials") pod "payload-processing-external-model-65665d6cd9-76q84" (UID: "e17326d8-7c07-4e33-b805-6287920734e5") : secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:36.187999 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:36.187966    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4e5279dc91ae: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:36.187999 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:36.187998    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65665d6cd9-76q84: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:36.188227 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:36.188071    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials podName:e17326d8-7c07-4e33-b805-6287920734e5 nodeName:}" failed. No retries permitted until 2026-11-03 12:09:37.188052338 +0000 UTC m=+1118.759636232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials") pod "payload-processing-external-model-65665d6cd9-76q84" (UID: "e17326d8-7c07-4e33-b805-6287920734e5") : secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:37.199911 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:37.199881    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4e5279dc91ae: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:37.199911 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:37.199910    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65665d6cd9-76q84: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:37.200379 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:37.200028    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials podName:e17326d8-7c07-4e33-b805-6287920734e5 nodeName:}" failed. No retries permitted until 2026-11-03 12:09:39.20000462 +0000 UTC m=+1120.771588527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials") pod "payload-processing-external-model-65665d6cd9-76q84" (UID: "e17326d8-7c07-4e33-b805-6287920734e5") : secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:37.654029 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:37.653942    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:09:39.227378 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:39.227339    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4e5279dc91ae: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:39.227378 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:39.227379    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65665d6cd9-76q84: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:39.227996 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:39.227467    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials podName:e17326d8-7c07-4e33-b805-6287920734e5 nodeName:}" failed. No retries permitted until 2026-11-03 12:09:43.227446707 +0000 UTC m=+1124.799030597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials") pod "payload-processing-external-model-65665d6cd9-76q84" (UID: "e17326d8-7c07-4e33-b805-6287920734e5") : secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:41.196108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:41.196069    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:09:43.230771 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:43.230709    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4e5279dc91ae: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:43.230771 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:43.230760    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65665d6cd9-76q84: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:43.231216 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:43.230819    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials podName:e17326d8-7c07-4e33-b805-6287920734e5 nodeName:}" failed. No retries permitted until 2026-11-03 12:09:51.230804815 +0000 UTC m=+1132.802388705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials") pod "payload-processing-external-model-65665d6cd9-76q84" (UID: "e17326d8-7c07-4e33-b805-6287920734e5") : secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:48.838283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:48.838247    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"]
Nov 03 12:09:48.839517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:48.839490    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:09:51.330282 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:51.330202    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4e5279dc91ae: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:51.330282 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:51.330233    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65665d6cd9-76q84: secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:51.330697 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:09:51.330292    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials podName:e17326d8-7c07-4e33-b805-6287920734e5 nodeName:}" failed. No retries permitted until 2026-11-03 12:10:07.330277083 +0000 UTC m=+1148.901860974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials") pod "payload-processing-external-model-65665d6cd9-76q84" (UID: "e17326d8-7c07-4e33-b805-6287920734e5") : secret "agc-cred-4e5279dc91ae" not found
Nov 03 12:09:52.682775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.681831    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"]
Nov 03 12:09:52.691235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.691211    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.979266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.979228    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.979629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.979606    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.979920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.979897    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.980178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.980161    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.980433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.980414    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6b2\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.980674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.980654    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.980923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.980905    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.981149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.981121    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.981375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.981357    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:52.984615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:52.984595    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"]
Nov 03 12:09:53.084378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.084334    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.085097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.085063    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.086060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.086039    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6b2\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.086594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.086572    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.087097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.085897    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.087246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.086991    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.087385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.085004    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.087557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.087537    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.092787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.088157    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.093560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.093264    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.093817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.093188    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.093817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.093541    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.094332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.094311    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.094512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.094495    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.100098 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.100048    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.113867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.113837    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.142931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.142873    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6b2\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.148134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.148083    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.315050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.314726    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:53.862826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:53.862019    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerName="payload-processing" containerID="cri-o://81f7f6f499c328e8a46cc6ef1ac11a3c5796816a04509d4b48ea7319b6ad2b1b" gracePeriod=25
Nov 03 12:09:54.232351 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.232313    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"]
Nov 03 12:09:54.292971 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:09:54.292915    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed6b079_a27d_4a6d_a49d_52cefe98ee71.slice/crio-c107b52a83bf3c0423509098910752e4980729adb5cdea7fe88b98c30d19890e WatchSource:0}: Error finding container c107b52a83bf3c0423509098910752e4980729adb5cdea7fe88b98c30d19890e: Status 404 returned error can't find the container with id c107b52a83bf3c0423509098910752e4980729adb5cdea7fe88b98c30d19890e
Nov 03 12:09:54.314621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.313870    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:09:54.314621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.313963    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:09:54.314621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.314007    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:09:54.417865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.405939    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerID="81f7f6f499c328e8a46cc6ef1ac11a3c5796816a04509d4b48ea7319b6ad2b1b" exitCode=0
Nov 03 12:09:54.417865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.406110    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" event={"ID":"6d58e22e-2b44-4736-bdfd-8d5924b04317","Type":"ContainerDied","Data":"81f7f6f499c328e8a46cc6ef1ac11a3c5796816a04509d4b48ea7319b6ad2b1b"}
Nov 03 12:09:54.421270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.421196    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" event={"ID":"7ed6b079-a27d-4a6d-a49d-52cefe98ee71","Type":"ContainerStarted","Data":"c107b52a83bf3c0423509098910752e4980729adb5cdea7fe88b98c30d19890e"}
Nov 03 12:09:54.649773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.647072    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:54.893776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:54.818731    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 12:09:55.271777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.252113    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:09:55.274555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.272324    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.274555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.272740    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.319240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.319200    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Nov 03 12:09:55.346938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.321834    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.347441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.322264    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded"
Nov 03 12:09:55.507851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.507821    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:55.508681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.508650    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl" event={"ID":"6d58e22e-2b44-4736-bdfd-8d5924b04317","Type":"ContainerDied","Data":"7f9ebbcc0ee59c09e8b55711e1482c471cd7d3a7647fe1fb4c4707e9c6a6e19d"}
Nov 03 12:09:55.508921 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.508905    4546 scope.go:122] "RemoveContainer" containerID="81f7f6f499c328e8a46cc6ef1ac11a3c5796816a04509d4b48ea7319b6ad2b1b"
Nov 03 12:09:55.513050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.513022    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" event={"ID":"7ed6b079-a27d-4a6d-a49d-52cefe98ee71","Type":"ContainerStarted","Data":"a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c"}
Nov 03 12:09:55.586156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.586118    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume\") pod \"6d58e22e-2b44-4736-bdfd-8d5924b04317\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") "
Nov 03 12:09:55.587892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.587863    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials\") pod \"6d58e22e-2b44-4736-bdfd-8d5924b04317\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") "
Nov 03 12:09:55.588729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.588708    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5\") pod \"6d58e22e-2b44-4736-bdfd-8d5924b04317\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") "
Nov 03 12:09:55.589021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.589000    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume\") pod \"6d58e22e-2b44-4736-bdfd-8d5924b04317\" (UID: \"6d58e22e-2b44-4736-bdfd-8d5924b04317\") "
Nov 03 12:09:55.590511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.588245    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume" pod "6d58e22e-2b44-4736-bdfd-8d5924b04317" (UID: "6d58e22e-2b44-4736-bdfd-8d5924b04317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:09:55.591444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.591422    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume" pod "6d58e22e-2b44-4736-bdfd-8d5924b04317" (UID: "6d58e22e-2b44-4736-bdfd-8d5924b04317"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:09:55.614035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.614003    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials" pod "6d58e22e-2b44-4736-bdfd-8d5924b04317" (UID: "6d58e22e-2b44-4736-bdfd-8d5924b04317"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:09:55.644319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.644281    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5" pod "6d58e22e-2b44-4736-bdfd-8d5924b04317" (UID: "6d58e22e-2b44-4736-bdfd-8d5924b04317"). InnerVolumeSpecName "kube-api-access-sj2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:09:55.677776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.650133    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.687775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.678739    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:09:55.725621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.721013    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj2r5\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-kube-api-access-sj2r5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:55.725621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.721048    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:55.725621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.721069    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58e22e-2b44-4736-bdfd-8d5924b04317-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:55.725621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:55.721084    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d58e22e-2b44-4736-bdfd-8d5924b04317-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:09:56.048561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.045893    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podStartSLOduration=4.04587158 podStartE2EDuration="4.04587158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:55.852136187 +0000 UTC m=+1137.423720128" watchObservedRunningTime="2026-11-03 12:09:56.04587158 +0000 UTC m=+1137.617455492"
Nov 03 12:09:56.130797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.127805    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:09:56.331832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.326433    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:09:56.331832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.329846    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:09:56.331832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.329916    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:09:56.564435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.564403    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"
Nov 03 12:09:56.740848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.740804    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"]
Nov 03 12:09:56.741181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:56.741146    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-h9jrl"]
Nov 03 12:09:57.331783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:57.322200    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:09:57.332332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:57.331898    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:09:58.331780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:58.330153    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:09:58.331780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:58.330232    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:09:58.577416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:58.577380    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" path="/var/lib/kubelet/pods/6d58e22e-2b44-4736-bdfd-8d5924b04317/volumes"
Nov 03 12:09:58.739308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:58.739276    4546 scope.go:122] "RemoveContainer" containerID="11dcdc3956bcbdeefa619c739ef9f08d8c0674a7418f5f07f9b871236f551987"
Nov 03 12:09:59.327775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:59.327248    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:09:59.327775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:09:59.327312    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:10:00.328321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:00.328285    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:10:00.330321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:00.328976    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:10:01.157236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:01.157200    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"]
Nov 03 12:10:01.317290 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:01.317243    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:10:01.317610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:01.317581    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:10:02.315982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:02.315944    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 12:10:02.316388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:02.316009    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 12:10:03.320020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:03.319989    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:10:03.320535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:03.320380    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:10:03.321486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:03.321438    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:10:05.281824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.281795    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"]
Nov 03 12:10:05.282337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.282322    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerName="payload-processing"
Nov 03 12:10:05.282392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.282338    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerName="payload-processing"
Nov 03 12:10:05.282447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.282438    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d58e22e-2b44-4736-bdfd-8d5924b04317" containerName="payload-processing"
Nov 03 12:10:05.287288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.287270    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.335898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.335865    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"kube-root-ca.crt\""
Nov 03 12:10:05.381924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.381897    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"openshift-service-ca.crt\""
Nov 03 12:10:05.382066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.381947    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"default-dockercfg-lk5jw\""
Nov 03 12:10:05.382066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.381957    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"distinct-2-w1-5951bc-kserve-self-signed-certs\""
Nov 03 12:10:05.385285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.385266    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"]
Nov 03 12:10:05.422045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.422009    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.422226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.422194    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.422315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.422268    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.422373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.422345    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.422428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.422395    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.422486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.422455    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.523813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.523785    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.523914    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524017    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524077    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524122    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524189    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524304    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524437    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.524538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.524521    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.526117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.526094    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.526481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.526466    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.576901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.576846    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f\") pod \"distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") " pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.598809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.598781    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:05.994243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:05.994219    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"]
Nov 03 12:10:05.994808 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:10:05.994785    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda64e6e11_031b_46be_a282_657483c47b7a.slice/crio-1c5f70a199ade095170bb8661515056fe0e400170ccdec7f38fc0e3f3e02d15a WatchSource:0}: Error finding container 1c5f70a199ade095170bb8661515056fe0e400170ccdec7f38fc0e3f3e02d15a: Status 404 returned error can't find the container with id 1c5f70a199ade095170bb8661515056fe0e400170ccdec7f38fc0e3f3e02d15a
Nov 03 12:10:06.158259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.158220    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84" podUID="e17326d8-7c07-4e33-b805-6287920734e5" containerName="payload-processing" containerID="cri-o://012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116" gracePeriod=25
Nov 03 12:10:06.404110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.404085    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:10:06.536905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.536878    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume\") pod \"e17326d8-7c07-4e33-b805-6287920734e5\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") "
Nov 03 12:10:06.537918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.536991    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume\") pod \"e17326d8-7c07-4e33-b805-6287920734e5\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") "
Nov 03 12:10:06.537918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.537054    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials\") pod \"e17326d8-7c07-4e33-b805-6287920734e5\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") "
Nov 03 12:10:06.537918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.537090    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr\") pod \"e17326d8-7c07-4e33-b805-6287920734e5\" (UID: \"e17326d8-7c07-4e33-b805-6287920734e5\") "
Nov 03 12:10:06.537918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.537410    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume" pod "e17326d8-7c07-4e33-b805-6287920734e5" (UID: "e17326d8-7c07-4e33-b805-6287920734e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:10:06.537918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.537635    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:06.537918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.537663    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume" pod "e17326d8-7c07-4e33-b805-6287920734e5" (UID: "e17326d8-7c07-4e33-b805-6287920734e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:10:06.540145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.540116    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials" pod "e17326d8-7c07-4e33-b805-6287920734e5" (UID: "e17326d8-7c07-4e33-b805-6287920734e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:10:06.540809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.540780    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr" pod "e17326d8-7c07-4e33-b805-6287920734e5" (UID: "e17326d8-7c07-4e33-b805-6287920734e5"). InnerVolumeSpecName "kube-api-access-bfdtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:10:06.638950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.638922    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17326d8-7c07-4e33-b805-6287920734e5-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:06.639096 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.638954    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:06.639096 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.638970    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdtr\" (UniqueName: \"kubernetes.io/projected/e17326d8-7c07-4e33-b805-6287920734e5-kube-api-access-bfdtr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:06.666587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.666557    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" event={"ID":"a64e6e11-031b-46be-a282-657483c47b7a","Type":"ContainerStarted","Data":"f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131"}
Nov 03 12:10:06.666712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.666603    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" event={"ID":"a64e6e11-031b-46be-a282-657483c47b7a","Type":"ContainerStarted","Data":"1c5f70a199ade095170bb8661515056fe0e400170ccdec7f38fc0e3f3e02d15a"}
Nov 03 12:10:06.667869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.667847    4546 generic.go:356] "Generic (PLEG): container finished" podID="e17326d8-7c07-4e33-b805-6287920734e5" containerID="012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116" exitCode=0
Nov 03 12:10:06.667980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.667885    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"
Nov 03 12:10:06.667980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.667932    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84" event={"ID":"e17326d8-7c07-4e33-b805-6287920734e5","Type":"ContainerDied","Data":"012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116"}
Nov 03 12:10:06.667980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.667975    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65665d6cd9-76q84" event={"ID":"e17326d8-7c07-4e33-b805-6287920734e5","Type":"ContainerDied","Data":"1d905aca205c357179bc54f4f4cb06c36bb553ba5f0d1cd7f398db4de150ebfe"}
Nov 03 12:10:06.668101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.667998    4546 scope.go:122] "RemoveContainer" containerID="012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116"
Nov 03 12:10:06.677080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.677065    4546 scope.go:122] "RemoveContainer" containerID="012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116"
Nov 03 12:10:06.677325 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:10:06.677310    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116\": container with ID starting with 012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116 not found: ID does not exist" containerID="012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116"
Nov 03 12:10:06.677371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.677332    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116"} err="failed to get container status \"012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116\": rpc error: code = NotFound desc = could not find container \"012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116\": container with ID starting with 012b7440d2aed1b5152fc0734afdbfa833a33e969936e3b7ed305036e4828116 not found: ID does not exist"
Nov 03 12:10:06.773435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.773384    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podStartSLOduration=1.773369285 podStartE2EDuration="1.773369285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:10:06.771550911 +0000 UTC m=+1148.343134827" watchObservedRunningTime="2026-11-03 12:10:06.773369285 +0000 UTC m=+1148.344953198"
Nov 03 12:10:06.874714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.874655    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"]
Nov 03 12:10:06.878297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:06.878277    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65665d6cd9-76q84"]
Nov 03 12:10:07.599006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:07.598964    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:07.650292 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:10:07.650257    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:10:08.573152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:08.573118    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17326d8-7c07-4e33-b805-6287920734e5" path="/var/lib/kubelet/pods/e17326d8-7c07-4e33-b805-6287920734e5/volumes"
Nov 03 12:10:15.601666 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:15.601632    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:15.713026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:15.712994    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:15.799776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:15.799735    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:10:16.372775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:16.371862    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:10:16.372775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:16.371976    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:10:17.031211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:17.031177    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:10:17.031864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:17.031828    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:10:32.716105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.716070    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"]
Nov 03 12:10:32.718669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.716599    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e17326d8-7c07-4e33-b805-6287920734e5" containerName="payload-processing"
Nov 03 12:10:32.718669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.716611    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17326d8-7c07-4e33-b805-6287920734e5" containerName="payload-processing"
Nov 03 12:10:32.718669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.716719    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17326d8-7c07-4e33-b805-6287920734e5" containerName="payload-processing"
Nov 03 12:10:32.720014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.719997    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.814302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.814267    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"]
Nov 03 12:10:32.833790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.833726    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.833968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.833844    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.833968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.833918    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp99l\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.833968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.833946    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.834155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.834103    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.834214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.834164    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.834214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.834205    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.834320 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.834232    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.834320 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.834280    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935257    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935338    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935403    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935441    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935470    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935512    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935572    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935608    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935687    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp99l\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935728    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935854    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.935990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.935970    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.936074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.936056    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.936211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.936191    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.938120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.938096    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.938254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.938237    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.990056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.989990    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:32.990056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:32.990030    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp99l\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:33.032945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.032915    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:33.449314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.449283    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"]
Nov 03 12:10:33.449461 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:10:33.449438    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a095a5f_8680_4b3c_a196_5db07230879f.slice/crio-3a65ab7b27c1f5da9c54734974240d2dc2f9a520d04d7712777c6cb3fb5157de WatchSource:0}: Error finding container 3a65ab7b27c1f5da9c54734974240d2dc2f9a520d04d7712777c6cb3fb5157de: Status 404 returned error can't find the container with id 3a65ab7b27c1f5da9c54734974240d2dc2f9a520d04d7712777c6cb3fb5157de
Nov 03 12:10:33.451969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.451931    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:10:33.452077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.452011    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:10:33.452077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.452056    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:10:33.874937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.874896    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" event={"ID":"0a095a5f-8680-4b3c-a196-5db07230879f","Type":"ContainerStarted","Data":"3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f"}
Nov 03 12:10:33.875331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.874968    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" event={"ID":"0a095a5f-8680-4b3c-a196-5db07230879f","Type":"ContainerStarted","Data":"3a65ab7b27c1f5da9c54734974240d2dc2f9a520d04d7712777c6cb3fb5157de"}
Nov 03 12:10:33.984508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:33.984444    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podStartSLOduration=1.984429173 podStartE2EDuration="1.984429173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:10:33.98247521 +0000 UTC m=+1175.554059133" watchObservedRunningTime="2026-11-03 12:10:33.984429173 +0000 UTC m=+1175.556013085"
Nov 03 12:10:34.033474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:34.033434    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:34.035257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:34.035227    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:34.035394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:34.035287    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:35.033696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:35.033659    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:35.034271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:35.033726    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:36.033788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:36.033739    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:36.034169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:36.033816    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:37.034239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:37.034207    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:37.034802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:37.034263    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:37.658568 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:10:37.658532    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:10:38.034228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:38.034194    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:38.034412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:38.034252    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:39.034132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:39.034097    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:39.034345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:39.034156    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:40.033476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:40.033439    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 12:10:40.033888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:40.033506    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 12:10:41.037840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:41.037810    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:41.038332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:41.038162    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:41.038728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:41.038707    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:10:43.610318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.610282    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"]
Nov 03 12:10:43.614124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.614106    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.664283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.664255    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:10:43.689701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.689675    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndmk\" (UniqueName: \"kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.689880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.689774    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.689974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.689955    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.709820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.709794    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:10:43.709820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.709816    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:10:43.709976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.709815    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:10:43.713377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.713357    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"]
Nov 03 12:10:43.791455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.791431    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndmk\" (UniqueName: \"kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.791640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.791527    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.791711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.791643    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.792221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.792196    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.794147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.794127    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.843919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.843893    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndmk\" (UniqueName: \"kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk\") pod \"maas-api-revoke-sub-7738265a-8kztq\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:43.928055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:43.927985    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:44.297032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:44.297001    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"]
Nov 03 12:10:44.298229 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:10:44.298187    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6588442e_0477_4d80_93b9_5a103f47bd0d.slice/crio-345a7cfb1dd1000f79f2d27a61b8388d872a2d5cec93cb179fbc0be749273732 WatchSource:0}: Error finding container 345a7cfb1dd1000f79f2d27a61b8388d872a2d5cec93cb179fbc0be749273732: Status 404 returned error can't find the container with id 345a7cfb1dd1000f79f2d27a61b8388d872a2d5cec93cb179fbc0be749273732
Nov 03 12:10:44.948339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:44.948252    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq" event={"ID":"6588442e-0477-4d80-93b9-5a103f47bd0d","Type":"ContainerStarted","Data":"e23489659d819e85a0b413c881973b6f826b4b11d3d24d226f3d5b0f81baa14c"}
Nov 03 12:10:44.948339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:44.948308    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq" event={"ID":"6588442e-0477-4d80-93b9-5a103f47bd0d","Type":"ContainerStarted","Data":"345a7cfb1dd1000f79f2d27a61b8388d872a2d5cec93cb179fbc0be749273732"}
Nov 03 12:10:45.056241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:45.056189    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq" podStartSLOduration=2.056175223 podStartE2EDuration="2.056175223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:10:45.053160657 +0000 UTC m=+1186.624744586" watchObservedRunningTime="2026-11-03 12:10:45.056175223 +0000 UTC m=+1186.627759134"
Nov 03 12:10:45.956301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:45.956266    4546 generic.go:356] "Generic (PLEG): container finished" podID="6588442e-0477-4d80-93b9-5a103f47bd0d" containerID="e23489659d819e85a0b413c881973b6f826b4b11d3d24d226f3d5b0f81baa14c" exitCode=0
Nov 03 12:10:45.956788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:45.956345    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq" event={"ID":"6588442e-0477-4d80-93b9-5a103f47bd0d","Type":"ContainerDied","Data":"e23489659d819e85a0b413c881973b6f826b4b11d3d24d226f3d5b0f81baa14c"}
Nov 03 12:10:47.118277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.118255    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:47.236730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.234939    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca\") pod \"6588442e-0477-4d80-93b9-5a103f47bd0d\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") "
Nov 03 12:10:47.236730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.234991    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk\" (UniqueName: \"kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk\") pod \"6588442e-0477-4d80-93b9-5a103f47bd0d\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") "
Nov 03 12:10:47.236730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.235040    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca\") pod \"6588442e-0477-4d80-93b9-5a103f47bd0d\" (UID: \"6588442e-0477-4d80-93b9-5a103f47bd0d\") "
Nov 03 12:10:47.236730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.235680    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca" pod "6588442e-0477-4d80-93b9-5a103f47bd0d" (UID: "6588442e-0477-4d80-93b9-5a103f47bd0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:10:47.238071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.237806    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk" pod "6588442e-0477-4d80-93b9-5a103f47bd0d" (UID: "6588442e-0477-4d80-93b9-5a103f47bd0d"). InnerVolumeSpecName "kube-api-access-6ndmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:10:47.238441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.238408    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca" pod "6588442e-0477-4d80-93b9-5a103f47bd0d" (UID: "6588442e-0477-4d80-93b9-5a103f47bd0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:10:47.336644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.336614    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:47.336644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.336642    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndmk\" (UniqueName: \"kubernetes.io/projected/6588442e-0477-4d80-93b9-5a103f47bd0d-kube-api-access-6ndmk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:47.336644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.336654    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588442e-0477-4d80-93b9-5a103f47bd0d-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:10:47.969917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.969885    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq" event={"ID":"6588442e-0477-4d80-93b9-5a103f47bd0d","Type":"ContainerDied","Data":"345a7cfb1dd1000f79f2d27a61b8388d872a2d5cec93cb179fbc0be749273732"}
Nov 03 12:10:47.969917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.969920    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"
Nov 03 12:10:47.970141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:47.969930    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345a7cfb1dd1000f79f2d27a61b8388d872a2d5cec93cb179fbc0be749273732"
Nov 03 12:10:54.079773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:54.078814    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 12:10:54.191775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:54.180818    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:10:58.767953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:58.767925    4546 scope.go:122] "RemoveContainer" containerID="7af54fa2c812603c4dd0112f7b5e3e2c2a9c959e8b625bca9a007160608b1fd7"
Nov 03 12:10:58.777288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:58.777267    4546 scope.go:122] "RemoveContainer" containerID="8c61fc924d38d0c21f0a484ae4b4e491e9cf0a72c014d8746fc07987e40c092a"
Nov 03 12:10:58.785829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:58.785811    4546 scope.go:122] "RemoveContainer" containerID="c809ee5a0f7b1cf4be2e17723d96f0e1e5a444ea02e44d200c0bf3dbd0ce4afd"
Nov 03 12:10:59.867534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.867499    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"]
Nov 03 12:10:59.868081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.868067    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6588442e-0477-4d80-93b9-5a103f47bd0d" containerName="revoke-keys"
Nov 03 12:10:59.868141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.868082    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6588442e-0477-4d80-93b9-5a103f47bd0d" containerName="revoke-keys"
Nov 03 12:10:59.868181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.868162    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6588442e-0477-4d80-93b9-5a103f47bd0d" containerName="revoke-keys"
Nov 03 12:10:59.873169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.873147    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:10:59.925062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.925031    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"]
Nov 03 12:10:59.937344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.937317    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcdk\" (UniqueName: \"kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:10:59.937484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.937359    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:10:59.937569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.937546    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:10:59.967225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.967193    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:10:59.967225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.967214    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:10:59.967386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.967216    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:10:59.967492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:10:59.967474    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:11:00.039155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.039126    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcdk\" (UniqueName: \"kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.039317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.039206    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.039375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.039354    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.040088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.040013    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.041693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.041671    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.092150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.092125    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcdk\" (UniqueName: \"kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk\") pod \"maas-api-revoke-sub-bc44345f-bwrs4\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.186193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.186112    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:00.559516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:00.559397    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"]
Nov 03 12:11:00.561047 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:00.561021    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba14fb1_fa7c_4d32_a530_0baee6adb556.slice/crio-0524eb0ed58fee91c4420806c95fabc05a6decb4e2530c435a15231d996e6cf7 WatchSource:0}: Error finding container 0524eb0ed58fee91c4420806c95fabc05a6decb4e2530c435a15231d996e6cf7: Status 404 returned error can't find the container with id 0524eb0ed58fee91c4420806c95fabc05a6decb4e2530c435a15231d996e6cf7
Nov 03 12:11:01.075353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:01.075254    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4" event={"ID":"7ba14fb1-fa7c-4d32-a530-0baee6adb556","Type":"ContainerStarted","Data":"791d027e4339992e79504f00206cb619767472895d6e0285e88640280ceb8406"}
Nov 03 12:11:01.075700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:01.075359    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4" event={"ID":"7ba14fb1-fa7c-4d32-a530-0baee6adb556","Type":"ContainerStarted","Data":"0524eb0ed58fee91c4420806c95fabc05a6decb4e2530c435a15231d996e6cf7"}
Nov 03 12:11:01.182918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:01.182874    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4" podStartSLOduration=2.18286009 podStartE2EDuration="2.18286009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:01.18018696 +0000 UTC m=+1202.751770899" watchObservedRunningTime="2026-11-03 12:11:01.18286009 +0000 UTC m=+1202.754444028"
Nov 03 12:11:02.082963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:02.082931    4546 generic.go:356] "Generic (PLEG): container finished" podID="7ba14fb1-fa7c-4d32-a530-0baee6adb556" containerID="791d027e4339992e79504f00206cb619767472895d6e0285e88640280ceb8406" exitCode=0
Nov 03 12:11:02.083364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:02.083017    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4" event={"ID":"7ba14fb1-fa7c-4d32-a530-0baee6adb556","Type":"ContainerDied","Data":"791d027e4339992e79504f00206cb619767472895d6e0285e88640280ceb8406"}
Nov 03 12:11:03.246963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.246939    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:03.280088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.280059    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca\") pod \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") "
Nov 03 12:11:03.280232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.280181    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk\" (UniqueName: \"kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk\") pod \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") "
Nov 03 12:11:03.280317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.280291    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca\") pod \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\" (UID: \"7ba14fb1-fa7c-4d32-a530-0baee6adb556\") "
Nov 03 12:11:03.280381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.280355    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca" pod "7ba14fb1-fa7c-4d32-a530-0baee6adb556" (UID: "7ba14fb1-fa7c-4d32-a530-0baee6adb556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:03.280794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.280744    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:03.282454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.282431    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca" pod "7ba14fb1-fa7c-4d32-a530-0baee6adb556" (UID: "7ba14fb1-fa7c-4d32-a530-0baee6adb556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:03.282565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.282544    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk" pod "7ba14fb1-fa7c-4d32-a530-0baee6adb556" (UID: "7ba14fb1-fa7c-4d32-a530-0baee6adb556"). InnerVolumeSpecName "kube-api-access-2gcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:03.382493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.382430    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcdk\" (UniqueName: \"kubernetes.io/projected/7ba14fb1-fa7c-4d32-a530-0baee6adb556-kube-api-access-2gcdk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:03.382493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:03.382454    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba14fb1-fa7c-4d32-a530-0baee6adb556-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:04.098855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:04.098829    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"
Nov 03 12:11:04.099031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:04.098826    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4" event={"ID":"7ba14fb1-fa7c-4d32-a530-0baee6adb556","Type":"ContainerDied","Data":"0524eb0ed58fee91c4420806c95fabc05a6decb4e2530c435a15231d996e6cf7"}
Nov 03 12:11:04.099031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:04.098943    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0524eb0ed58fee91c4420806c95fabc05a6decb4e2530c435a15231d996e6cf7"
Nov 03 12:11:07.744894 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:11:07.744606    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:11:15.637782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.637730    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"]
Nov 03 12:11:15.638494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.638473    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7ba14fb1-fa7c-4d32-a530-0baee6adb556" containerName="revoke-keys"
Nov 03 12:11:15.638591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.638497    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba14fb1-fa7c-4d32-a530-0baee6adb556" containerName="revoke-keys"
Nov 03 12:11:15.638672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.638658    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba14fb1-fa7c-4d32-a530-0baee6adb556" containerName="revoke-keys"
Nov 03 12:11:15.642036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.642014    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.691357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.691332    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:11:15.691357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.691347    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:11:15.737606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.737556    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:11:15.737800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.737723    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:11:15.741399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.741372    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"]
Nov 03 12:11:15.766682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.766658    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkm2\" (UniqueName: \"kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.766821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.766689    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.766958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.766938    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.811837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.811810    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"]
Nov 03 12:11:15.815914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.815894    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.868322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.868276    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.868460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.868384    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.868531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.868479    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.868605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.868582    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkm2\" (UniqueName: \"kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.868700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.868619    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.868700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.868681    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjgg\" (UniqueName: \"kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.869150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.869128    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.869814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.869790    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"]
Nov 03 12:11:15.871624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.871602    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.924594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.924528    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkm2\" (UniqueName: \"kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2\") pod \"maas-api-revoke-sub-88ac920c-kcvf4\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.955681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.955655    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:15.971292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.971269    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjgg\" (UniqueName: \"kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.971412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.971354    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.971482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.971449    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.972230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.972211    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:15.973958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:15.973939    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:16.038526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:16.038103    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjgg\" (UniqueName: \"kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg\") pod \"maas-api-revoke-sub-7f292f59-8z9fm\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:16.128891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:16.128861    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:16.325167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:16.325143    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"]
Nov 03 12:11:16.325630 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:16.325608    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56446e5e_cfe6_4e01_b743_917bd6e6b27e.slice/crio-367c68c3e1f40dd54711613aefe964c96cbf255c076538abb42773477fc51ab1 WatchSource:0}: Error finding container 367c68c3e1f40dd54711613aefe964c96cbf255c076538abb42773477fc51ab1: Status 404 returned error can't find the container with id 367c68c3e1f40dd54711613aefe964c96cbf255c076538abb42773477fc51ab1
Nov 03 12:11:16.496616 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:16.496589    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd904dc7_4d52_4833_a4ea_268e492b7a6c.slice/crio-27956daf71fafee3a24ff9868e00e04e49893756026a46c9a5cc299af0d4d06e WatchSource:0}: Error finding container 27956daf71fafee3a24ff9868e00e04e49893756026a46c9a5cc299af0d4d06e: Status 404 returned error can't find the container with id 27956daf71fafee3a24ff9868e00e04e49893756026a46c9a5cc299af0d4d06e
Nov 03 12:11:16.496723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:16.496615    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"]
Nov 03 12:11:17.225009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:17.224976    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm" event={"ID":"dd904dc7-4d52-4833-a4ea-268e492b7a6c","Type":"ContainerStarted","Data":"0e773aaba3cb59582edd85bee5a0449ca5b125b20d492a0e253715cb18570d3a"}
Nov 03 12:11:17.225432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:17.225069    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm" event={"ID":"dd904dc7-4d52-4833-a4ea-268e492b7a6c","Type":"ContainerStarted","Data":"27956daf71fafee3a24ff9868e00e04e49893756026a46c9a5cc299af0d4d06e"}
Nov 03 12:11:17.227224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:17.227200    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4" event={"ID":"56446e5e-cfe6-4e01-b743-917bd6e6b27e","Type":"ContainerStarted","Data":"3bec065e7cc6b201e752f56ad59a4a27dc759a78c05b678c4bfa5eabac83f795"}
Nov 03 12:11:17.227346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:17.227241    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4" event={"ID":"56446e5e-cfe6-4e01-b743-917bd6e6b27e","Type":"ContainerStarted","Data":"367c68c3e1f40dd54711613aefe964c96cbf255c076538abb42773477fc51ab1"}
Nov 03 12:11:17.333299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:17.333248    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm" podStartSLOduration=2.33323456 podStartE2EDuration="2.33323456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:17.329664534 +0000 UTC m=+1218.901248466" watchObservedRunningTime="2026-11-03 12:11:17.33323456 +0000 UTC m=+1218.904818520"
Nov 03 12:11:17.439433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:17.439381    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4" podStartSLOduration=2.439366856 podStartE2EDuration="2.439366856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:17.435727133 +0000 UTC m=+1219.007311057" watchObservedRunningTime="2026-11-03 12:11:17.439366856 +0000 UTC m=+1219.010950768"
Nov 03 12:11:18.070827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.070783    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"]
Nov 03 12:11:18.075493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.075472    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.198547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.198505    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.198858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.198833    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbl9\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199054    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199132    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199183    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199210    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199272    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199298    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.199469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.199399    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.234783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.234713    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd904dc7-4d52-4833-a4ea-268e492b7a6c" containerID="0e773aaba3cb59582edd85bee5a0449ca5b125b20d492a0e253715cb18570d3a" exitCode=0
Nov 03 12:11:18.235240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.234812    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm" event={"ID":"dd904dc7-4d52-4833-a4ea-268e492b7a6c","Type":"ContainerDied","Data":"0e773aaba3cb59582edd85bee5a0449ca5b125b20d492a0e253715cb18570d3a"}
Nov 03 12:11:18.236573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.236537    4546 generic.go:356] "Generic (PLEG): container finished" podID="56446e5e-cfe6-4e01-b743-917bd6e6b27e" containerID="3bec065e7cc6b201e752f56ad59a4a27dc759a78c05b678c4bfa5eabac83f795" exitCode=0
Nov 03 12:11:18.236692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.236612    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4" event={"ID":"56446e5e-cfe6-4e01-b743-917bd6e6b27e","Type":"ContainerDied","Data":"3bec065e7cc6b201e752f56ad59a4a27dc759a78c05b678c4bfa5eabac83f795"}
Nov 03 12:11:18.266803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.266769    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"]
Nov 03 12:11:18.300374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300346    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300381    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300465    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300532    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300566    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbl9\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300649    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300713    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300800    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300817    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.300875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300838    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.301104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.300927    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.301196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.301172    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.301261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.301207    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.301555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.301532    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.303662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.303638    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.303775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.303692    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.353518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.353445    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.353678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.353579    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbl9\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.388655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.388626    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:18.800930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.800896    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"]
Nov 03 12:11:18.801311 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:18.801286    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d16bfbf_11ea_4e78_8177_7b5771944e3c.slice/crio-275b5233a053dd8ff41edc97805187c0e051a12c1147018e9d102dc76fb0deb3 WatchSource:0}: Error finding container 275b5233a053dd8ff41edc97805187c0e051a12c1147018e9d102dc76fb0deb3: Status 404 returned error can't find the container with id 275b5233a053dd8ff41edc97805187c0e051a12c1147018e9d102dc76fb0deb3
Nov 03 12:11:18.803742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.803713    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:11:18.803849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.803811    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:11:18.803849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:18.803843    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:11:19.250548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.250471    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" event={"ID":"2d16bfbf-11ea-4e78-8177-7b5771944e3c","Type":"ContainerStarted","Data":"e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc"}
Nov 03 12:11:19.250548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.250537    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" event={"ID":"2d16bfbf-11ea-4e78-8177-7b5771944e3c","Type":"ContainerStarted","Data":"275b5233a053dd8ff41edc97805187c0e051a12c1147018e9d102dc76fb0deb3"}
Nov 03 12:11:19.365532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.365464    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podStartSLOduration=1.365449085 podStartE2EDuration="1.365449085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:19.361771168 +0000 UTC m=+1220.933355088" watchObservedRunningTime="2026-11-03 12:11:19.365449085 +0000 UTC m=+1220.937032996"
Nov 03 12:11:19.389132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.389078    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:19.394520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.394056    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:19.394520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.394120    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:19.482951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.482926    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:19.486929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.486910    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:19.634064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.633964    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2\" (UniqueName: \"kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2\") pod \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") "
Nov 03 12:11:19.634064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.634037    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca\") pod \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") "
Nov 03 12:11:19.634297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.634104    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca\") pod \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") "
Nov 03 12:11:19.634297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.634205    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca\") pod \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\" (UID: \"56446e5e-cfe6-4e01-b743-917bd6e6b27e\") "
Nov 03 12:11:19.634410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.634304    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca\") pod \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") "
Nov 03 12:11:19.634410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.634381    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg\" (UniqueName: \"kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg\") pod \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\" (UID: \"dd904dc7-4d52-4833-a4ea-268e492b7a6c\") "
Nov 03 12:11:19.641049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.638809    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca" pod "dd904dc7-4d52-4833-a4ea-268e492b7a6c" (UID: "dd904dc7-4d52-4833-a4ea-268e492b7a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:19.641049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.639262    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca" pod "56446e5e-cfe6-4e01-b743-917bd6e6b27e" (UID: "56446e5e-cfe6-4e01-b743-917bd6e6b27e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:19.641049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.639332    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca" pod "dd904dc7-4d52-4833-a4ea-268e492b7a6c" (UID: "dd904dc7-4d52-4833-a4ea-268e492b7a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:19.641049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.639369    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2" pod "56446e5e-cfe6-4e01-b743-917bd6e6b27e" (UID: "56446e5e-cfe6-4e01-b743-917bd6e6b27e"). InnerVolumeSpecName "kube-api-access-wwkm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:19.647977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.647946    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg" pod "dd904dc7-4d52-4833-a4ea-268e492b7a6c" (UID: "dd904dc7-4d52-4833-a4ea-268e492b7a6c"). InnerVolumeSpecName "kube-api-access-fsjgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:19.651032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.650848    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca" pod "56446e5e-cfe6-4e01-b743-917bd6e6b27e" (UID: "56446e5e-cfe6-4e01-b743-917bd6e6b27e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:19.736463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.736420    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:19.736636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.736475    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:19.736636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.736495    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd904dc7-4d52-4833-a4ea-268e492b7a6c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:19.736636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.736511    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsjgg\" (UniqueName: \"kubernetes.io/projected/dd904dc7-4d52-4833-a4ea-268e492b7a6c-kube-api-access-fsjgg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:19.736636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.736526    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwkm2\" (UniqueName: \"kubernetes.io/projected/56446e5e-cfe6-4e01-b743-917bd6e6b27e-kube-api-access-wwkm2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:19.736636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:19.736554    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56446e5e-cfe6-4e01-b743-917bd6e6b27e-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:20.258530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.258500    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"
Nov 03 12:11:20.259008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.258505    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm" event={"ID":"dd904dc7-4d52-4833-a4ea-268e492b7a6c","Type":"ContainerDied","Data":"27956daf71fafee3a24ff9868e00e04e49893756026a46c9a5cc299af0d4d06e"}
Nov 03 12:11:20.259008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.258650    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27956daf71fafee3a24ff9868e00e04e49893756026a46c9a5cc299af0d4d06e"
Nov 03 12:11:20.260000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.259977    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"
Nov 03 12:11:20.260111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.260008    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4" event={"ID":"56446e5e-cfe6-4e01-b743-917bd6e6b27e","Type":"ContainerDied","Data":"367c68c3e1f40dd54711613aefe964c96cbf255c076538abb42773477fc51ab1"}
Nov 03 12:11:20.260111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.260056    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="367c68c3e1f40dd54711613aefe964c96cbf255c076538abb42773477fc51ab1"
Nov 03 12:11:20.389841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.389808    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:20.390009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:20.389859    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:21.335504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.335463    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"]
Nov 03 12:11:21.336385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.336363    4546 cpu_manager.go:414] "containerMap: removing container" podUID="dd904dc7-4d52-4833-a4ea-268e492b7a6c" containerName="revoke-keys"
Nov 03 12:11:21.336456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.336391    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd904dc7-4d52-4833-a4ea-268e492b7a6c" containerName="revoke-keys"
Nov 03 12:11:21.336456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.336429    4546 cpu_manager.go:414] "containerMap: removing container" podUID="56446e5e-cfe6-4e01-b743-917bd6e6b27e" containerName="revoke-keys"
Nov 03 12:11:21.336456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.336438    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56446e5e-cfe6-4e01-b743-917bd6e6b27e" containerName="revoke-keys"
Nov 03 12:11:21.336610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.336593    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd904dc7-4d52-4833-a4ea-268e492b7a6c" containerName="revoke-keys"
Nov 03 12:11:21.336645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.336622    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="56446e5e-cfe6-4e01-b743-917bd6e6b27e" containerName="revoke-keys"
Nov 03 12:11:21.340675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.340655    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.389061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.389030    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:21.389203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.389078    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:21.437944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.437834    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"]
Nov 03 12:11:21.439942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.439914    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9893f0-openshift-default-dockercfg-ptlmk\""
Nov 03 12:11:21.455171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455147    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455188    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455207    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455322    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455370    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455401    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455440    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.455623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455595    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vld29\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.456010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.455672    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.556962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.556924    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.556962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.556966    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557078    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557114    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557169    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557207    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557262    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557331    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557335    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557602    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557726    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vld29\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557783    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557792    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.557944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.557890    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.559962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.559940    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.560143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.560127    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.638996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.638879    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.642442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.642407    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vld29\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:21.654144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:21.654115    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:22.102163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.102140    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"]
Nov 03 12:11:22.102627 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:22.102590    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bbea76_9808_42e8_af21_8b2c11cef1c6.slice/crio-81978602f988623fc26df75a44b4710a06b6111395301ae13db0663f69e2ae4b WatchSource:0}: Error finding container 81978602f988623fc26df75a44b4710a06b6111395301ae13db0663f69e2ae4b: Status 404 returned error can't find the container with id 81978602f988623fc26df75a44b4710a06b6111395301ae13db0663f69e2ae4b
Nov 03 12:11:22.105074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.105044    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:11:22.105137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.105110    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:11:22.105173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.105140    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:11:22.276725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.276692    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" event={"ID":"89bbea76-9808-42e8-af21-8b2c11cef1c6","Type":"ContainerStarted","Data":"b42f12542c3234e783f4e122f24da5a63588b634156277aa4bcc4c6852944045"}
Nov 03 12:11:22.276935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.276759    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" event={"ID":"89bbea76-9808-42e8-af21-8b2c11cef1c6","Type":"ContainerStarted","Data":"81978602f988623fc26df75a44b4710a06b6111395301ae13db0663f69e2ae4b"}
Nov 03 12:11:22.389946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.389854    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:22.389946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.389908    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:22.391617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.391572    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podStartSLOduration=1.391558826 podStartE2EDuration="1.391558826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:22.387091253 +0000 UTC m=+1223.958675178" watchObservedRunningTime="2026-11-03 12:11:22.391558826 +0000 UTC m=+1223.963142748"
Nov 03 12:11:22.655095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:22.654999    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:23.389103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:23.389071    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:23.389252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:23.389118    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:23.657181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:23.657078    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:23.657181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:23.657135    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:24.389887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:24.389854    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:24.390080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:24.389910    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:24.657296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:24.657202    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:24.657800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:24.657286    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:25.389337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:25.389299    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:11:25.389520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:25.389362    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:11:25.654803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:25.654703    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:25.654803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:25.654770    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:26.361141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.361104    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"]
Nov 03 12:11:26.365369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.365349    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.394260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.394228    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:26.394658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.394628    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:26.396103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.396069    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:11:26.421860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.421830    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:11:26.440218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.421965    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:11:26.445866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.445818    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.446230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.446203    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t5s\" (UniqueName: \"kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.446501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.446480    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.506264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.506232    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:11:26.506622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.506604    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:11:26.591784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.549891    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.624353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.624263    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t5s\" (UniqueName: \"kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.629095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.629045    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.639899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.635244    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:26.669777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.662856    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:26.669777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.662913    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:26.911933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:26.892734    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t5s\" (UniqueName: \"kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:27.123779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.118597    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"]
Nov 03 12:11:27.187775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.146220    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0aaf99e-tcpmj\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:27.302009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.301971    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:27.681907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.681867    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:27.682962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.682478    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:27.756946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.756900    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:27.759002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:27.757196    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:28.149948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.149845    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:11:28.176910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.176855    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:28.180864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.180820    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:28.238468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.238424    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:28.319914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.319869    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:28.751777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.747885    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:28.775805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.752386    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:28.789800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:28.770151    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.199156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.094033    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:29.244533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.242935    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.350537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.350490    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.434051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.433897    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.520790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.520124    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.557779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.523901    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.616985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.562917    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 12:11:29.617228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.562855    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 12:11:29.657517 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:11:29.657483    4546 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.036s"
Nov 03 12:11:29.686935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.686903    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:11:29.687335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.687316    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"]
Nov 03 12:11:29.688030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.688012    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:11:29.688574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.688549    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:11:29.688764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.688733    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:11:29.688944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.688920    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:11:29.689094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.689082    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:11:29.788845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.781284    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:29.815025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.814975    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:29.822714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.822662    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:29.917468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:29.917416    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj" event={"ID":"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4","Type":"ContainerStarted","Data":"dc5bc45d81b2179e8051f868ac8b2cacdf33e7268396f52bdccb8300e1be706d"}
Nov 03 12:11:30.465131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.465080    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:11:30.492709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.490531    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:11:30.545529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.543390    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:11:30.567297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.554338    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:11:30.567297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.562095    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:11:30.567297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.563407    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:11:30.655538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.655507    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:30.655666 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.655559    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:30.952177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:30.952087    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj" event={"ID":"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4","Type":"ContainerStarted","Data":"07e0998839d88e29a5dc15ce4b119e3316bad9972e817c1ae5b074aa2ce7a01c"}
Nov 03 12:11:31.550893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:31.550843    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj" podStartSLOduration=5.550826968 podStartE2EDuration="5.550826968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:31.54756935 +0000 UTC m=+1233.119153273" watchObservedRunningTime="2026-11-03 12:11:31.550826968 +0000 UTC m=+1233.122410879"
Nov 03 12:11:31.654808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:31.654727    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:11:31.654978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:31.654833    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 12:11:31.960271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:31.960190    4546 generic.go:356] "Generic (PLEG): container finished" podID="99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" containerID="07e0998839d88e29a5dc15ce4b119e3316bad9972e817c1ae5b074aa2ce7a01c" exitCode=0
Nov 03 12:11:31.960627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:31.960273    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj" event={"ID":"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4","Type":"ContainerDied","Data":"07e0998839d88e29a5dc15ce4b119e3316bad9972e817c1ae5b074aa2ce7a01c"}
Nov 03 12:11:32.634126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.634095    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"]
Nov 03 12:11:32.638571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.638549    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.658379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.658360    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:32.658639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.658614    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:32.659359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.659342    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:11:32.694900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.694872    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"]
Nov 03 12:11:32.699325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.699294    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.699426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.699404    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmg8\" (UniqueName: \"kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.699482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.699471    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.801463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.801431    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmg8\" (UniqueName: \"kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.801620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.801493    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.801806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.801735    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.802425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.802400    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.804045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.804026    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.856535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.856502    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmg8\" (UniqueName: \"kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8\") pod \"maas-api-revoke-sub-d684072e-nstls\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:32.951712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:32.951634    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:33.186301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.186265    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:33.309299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.309263    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca\") pod \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") "
Nov 03 12:11:33.309492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.309444    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca\") pod \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") "
Nov 03 12:11:33.309565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.309501    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s\" (UniqueName: \"kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s\") pod \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\" (UID: \"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4\") "
Nov 03 12:11:33.309881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.309855    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca" pod "99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" (UID: "99c35050-7a2a-4c2d-9bee-9ad19d1fbef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:33.310111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.310091    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:33.313906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.313871    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s" pod "99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" (UID: "99c35050-7a2a-4c2d-9bee-9ad19d1fbef4"). InnerVolumeSpecName "kube-api-access-x6t5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:33.314197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.314173    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca" pod "99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" (UID: "99c35050-7a2a-4c2d-9bee-9ad19d1fbef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:33.338563 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:33.338528    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91f7944_8518_4982_9545_a76252c09237.slice/crio-5c00f8e401f7cadb1744c06e152e79b8d37f5ca0e348108d52bd9bb4b2777d29 WatchSource:0}: Error finding container 5c00f8e401f7cadb1744c06e152e79b8d37f5ca0e348108d52bd9bb4b2777d29: Status 404 returned error can't find the container with id 5c00f8e401f7cadb1744c06e152e79b8d37f5ca0e348108d52bd9bb4b2777d29
Nov 03 12:11:33.339791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.339741    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"]
Nov 03 12:11:33.411348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.411323    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6t5s\" (UniqueName: \"kubernetes.io/projected/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-kube-api-access-x6t5s\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:33.411348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.411350    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:33.978663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.978623    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj" event={"ID":"99c35050-7a2a-4c2d-9bee-9ad19d1fbef4","Type":"ContainerDied","Data":"dc5bc45d81b2179e8051f868ac8b2cacdf33e7268396f52bdccb8300e1be706d"}
Nov 03 12:11:33.978879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.978711    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5bc45d81b2179e8051f868ac8b2cacdf33e7268396f52bdccb8300e1be706d"
Nov 03 12:11:33.978879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.978633    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"
Nov 03 12:11:33.980317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.980281    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls" event={"ID":"c91f7944-8518-4982-9545-a76252c09237","Type":"ContainerStarted","Data":"b5d370ce278f380e94993ef13c1ac8847535a7f9941eeda838378e31d54ddbf4"}
Nov 03 12:11:33.980444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:33.980330    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls" event={"ID":"c91f7944-8518-4982-9545-a76252c09237","Type":"ContainerStarted","Data":"5c00f8e401f7cadb1744c06e152e79b8d37f5ca0e348108d52bd9bb4b2777d29"}
Nov 03 12:11:34.308175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:34.308078    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls" podStartSLOduration=2.308062376 podStartE2EDuration="2.308062376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:34.085506358 +0000 UTC m=+1235.657090281" watchObservedRunningTime="2026-11-03 12:11:34.308062376 +0000 UTC m=+1235.879646289"
Nov 03 12:11:34.988680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:34.988651    4546 generic.go:356] "Generic (PLEG): container finished" podID="c91f7944-8518-4982-9545-a76252c09237" containerID="b5d370ce278f380e94993ef13c1ac8847535a7f9941eeda838378e31d54ddbf4" exitCode=0
Nov 03 12:11:34.988862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:34.988739    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls" event={"ID":"c91f7944-8518-4982-9545-a76252c09237","Type":"ContainerDied","Data":"b5d370ce278f380e94993ef13c1ac8847535a7f9941eeda838378e31d54ddbf4"}
Nov 03 12:11:36.155910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.155885    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:36.250331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.250228    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8\" (UniqueName: \"kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8\") pod \"c91f7944-8518-4982-9545-a76252c09237\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") "
Nov 03 12:11:36.250488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.250380    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca\") pod \"c91f7944-8518-4982-9545-a76252c09237\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") "
Nov 03 12:11:36.250488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.250411    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca\") pod \"c91f7944-8518-4982-9545-a76252c09237\" (UID: \"c91f7944-8518-4982-9545-a76252c09237\") "
Nov 03 12:11:36.250848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.250814    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca" pod "c91f7944-8518-4982-9545-a76252c09237" (UID: "c91f7944-8518-4982-9545-a76252c09237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:36.251048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.251028    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f7944-8518-4982-9545-a76252c09237-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:36.252945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.252914    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8" pod "c91f7944-8518-4982-9545-a76252c09237" (UID: "c91f7944-8518-4982-9545-a76252c09237"). InnerVolumeSpecName "kube-api-access-6nmg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:36.252945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.252937    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca" pod "c91f7944-8518-4982-9545-a76252c09237" (UID: "c91f7944-8518-4982-9545-a76252c09237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:36.352043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.352003    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nmg8\" (UniqueName: \"kubernetes.io/projected/c91f7944-8518-4982-9545-a76252c09237-kube-api-access-6nmg8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:36.352043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:36.352038    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f7944-8518-4982-9545-a76252c09237-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:37.001409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:37.001381    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"
Nov 03 12:11:37.001584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:37.001420    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls" event={"ID":"c91f7944-8518-4982-9545-a76252c09237","Type":"ContainerDied","Data":"5c00f8e401f7cadb1744c06e152e79b8d37f5ca0e348108d52bd9bb4b2777d29"}
Nov 03 12:11:37.001584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:37.001463    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c00f8e401f7cadb1744c06e152e79b8d37f5ca0e348108d52bd9bb4b2777d29"
Nov 03 12:11:37.650613 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:11:37.650576    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:11:41.075776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:41.075086    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"]
Nov 03 12:11:41.138780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:41.123318    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58bc805-4fqz2"]
Nov 03 12:11:42.573708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.573657    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c959724-1374-4817-ac2e-e32121f9a7f5" path="/var/lib/kubelet/pods/5c959724-1374-4817-ac2e-e32121f9a7f5/volumes"
Nov 03 12:11:42.626771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.626717    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"]
Nov 03 12:11:42.627276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.627262    4546 cpu_manager.go:414] "containerMap: removing container" podUID="99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" containerName="revoke-keys"
Nov 03 12:11:42.627336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.627278    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" containerName="revoke-keys"
Nov 03 12:11:42.627336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.627304    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c91f7944-8518-4982-9545-a76252c09237" containerName="revoke-keys"
Nov 03 12:11:42.627336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.627309    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91f7944-8518-4982-9545-a76252c09237" containerName="revoke-keys"
Nov 03 12:11:42.627423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.627404    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91f7944-8518-4982-9545-a76252c09237" containerName="revoke-keys"
Nov 03 12:11:42.627423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.627416    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" containerName="revoke-keys"
Nov 03 12:11:42.632018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.631998    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.684178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.684149    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"]
Nov 03 12:11:42.726002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.725974    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:11:42.726602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.726587    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:11:42.726655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.726629    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:11:42.726717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.726661    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:11:42.808632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.808602    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.808802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.808776    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.808802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.808800    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqwd8\" (UniqueName: \"kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.909860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.909777    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.909860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.909810    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqwd8\" (UniqueName: \"kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.910083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.909912    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.910513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.910492    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.912257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.912235    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:42.962502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:42.962480    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqwd8\" (UniqueName: \"kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8\") pod \"maas-api-revoke-sub-f1da2654-pwmc2\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:43.245976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.245948    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:43.344397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.344363    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"]
Nov 03 12:11:43.350534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.350504    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.399569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.399539    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"]
Nov 03 12:11:43.516865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.516784    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzg7\" (UniqueName: \"kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.516865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.516827    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.517020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.516948    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.613837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.613804    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"]
Nov 03 12:11:43.614326 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:43.614302    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7196841_9f2e_4cee_a435_509f70d34bea.slice/crio-dd72a07b6e92f5e48a05eda836050518dc5633a3f19076dc9fd1c0883907787a WatchSource:0}: Error finding container dd72a07b6e92f5e48a05eda836050518dc5633a3f19076dc9fd1c0883907787a: Status 404 returned error can't find the container with id dd72a07b6e92f5e48a05eda836050518dc5633a3f19076dc9fd1c0883907787a
Nov 03 12:11:43.617629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.617608    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzg7\" (UniqueName: \"kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.617709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.617640    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.617815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.617794    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.618416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.618351    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.619967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.619950    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.670261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.670237    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzg7\" (UniqueName: \"kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7\") pod \"maas-api-revoke-sub-d662fe04-nm7fj\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:43.971575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:43.971542    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:44.110627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:44.110542    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2" event={"ID":"e7196841-9f2e-4cee-a435-509f70d34bea","Type":"ContainerStarted","Data":"7de636e91cd1ae7d9ef1fbce92ca733a649c927d10d240cce1abbec582ece773"}
Nov 03 12:11:44.110627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:44.110614    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2" event={"ID":"e7196841-9f2e-4cee-a435-509f70d34bea","Type":"ContainerStarted","Data":"dd72a07b6e92f5e48a05eda836050518dc5633a3f19076dc9fd1c0883907787a"}
Nov 03 12:11:44.215954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:44.215902    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2" podStartSLOduration=2.215889603 podStartE2EDuration="2.215889603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:44.214726971 +0000 UTC m=+1245.786310906" watchObservedRunningTime="2026-11-03 12:11:44.215889603 +0000 UTC m=+1245.787473515"
Nov 03 12:11:44.343852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:44.343821    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"]
Nov 03 12:11:44.344333 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:11:44.344280    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1cbe70_1a7d_4b6f_9888_f3f1e36e7ba6.slice/crio-da95953d91dfe455a9cb3c20fcb7dfd5ff8879ad9663c1db74fd589740b75b46 WatchSource:0}: Error finding container da95953d91dfe455a9cb3c20fcb7dfd5ff8879ad9663c1db74fd589740b75b46: Status 404 returned error can't find the container with id da95953d91dfe455a9cb3c20fcb7dfd5ff8879ad9663c1db74fd589740b75b46
Nov 03 12:11:45.118354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:45.118309    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj" event={"ID":"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6","Type":"ContainerStarted","Data":"1c9f0600fc741859e8b9ad9ad50db927b52c989d07a2933f0e4144ab7ac92e1e"}
Nov 03 12:11:45.118820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:45.118376    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj" event={"ID":"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6","Type":"ContainerStarted","Data":"da95953d91dfe455a9cb3c20fcb7dfd5ff8879ad9663c1db74fd589740b75b46"}
Nov 03 12:11:45.120192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:45.120171    4546 generic.go:356] "Generic (PLEG): container finished" podID="e7196841-9f2e-4cee-a435-509f70d34bea" containerID="7de636e91cd1ae7d9ef1fbce92ca733a649c927d10d240cce1abbec582ece773" exitCode=0
Nov 03 12:11:45.120310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:45.120211    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2" event={"ID":"e7196841-9f2e-4cee-a435-509f70d34bea","Type":"ContainerDied","Data":"7de636e91cd1ae7d9ef1fbce92ca733a649c927d10d240cce1abbec582ece773"}
Nov 03 12:11:45.226229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:45.226184    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj" podStartSLOduration=2.226170128 podStartE2EDuration="2.226170128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:45.223891386 +0000 UTC m=+1246.795475324" watchObservedRunningTime="2026-11-03 12:11:45.226170128 +0000 UTC m=+1246.797754041"
Nov 03 12:11:46.128427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.128393    4546 generic.go:356] "Generic (PLEG): container finished" podID="2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" containerID="1c9f0600fc741859e8b9ad9ad50db927b52c989d07a2933f0e4144ab7ac92e1e" exitCode=0
Nov 03 12:11:46.128903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.128481    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj" event={"ID":"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6","Type":"ContainerDied","Data":"1c9f0600fc741859e8b9ad9ad50db927b52c989d07a2933f0e4144ab7ac92e1e"}
Nov 03 12:11:46.292566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.292538    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:46.453421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.453335    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8\" (UniqueName: \"kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8\") pod \"e7196841-9f2e-4cee-a435-509f70d34bea\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") "
Nov 03 12:11:46.453421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.453373    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca\") pod \"e7196841-9f2e-4cee-a435-509f70d34bea\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") "
Nov 03 12:11:46.453649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.453560    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca\") pod \"e7196841-9f2e-4cee-a435-509f70d34bea\" (UID: \"e7196841-9f2e-4cee-a435-509f70d34bea\") "
Nov 03 12:11:46.453945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.453914    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca" pod "e7196841-9f2e-4cee-a435-509f70d34bea" (UID: "e7196841-9f2e-4cee-a435-509f70d34bea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:46.455687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.455663    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca" pod "e7196841-9f2e-4cee-a435-509f70d34bea" (UID: "e7196841-9f2e-4cee-a435-509f70d34bea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:46.455834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.455723    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8" pod "e7196841-9f2e-4cee-a435-509f70d34bea" (UID: "e7196841-9f2e-4cee-a435-509f70d34bea"). InnerVolumeSpecName "kube-api-access-jqwd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:46.555602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.555571    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:46.555602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.555599    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqwd8\" (UniqueName: \"kubernetes.io/projected/e7196841-9f2e-4cee-a435-509f70d34bea-kube-api-access-jqwd8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:46.555917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:46.555610    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7196841-9f2e-4cee-a435-509f70d34bea-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:47.136058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.136032    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"
Nov 03 12:11:47.136569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.136065    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2" event={"ID":"e7196841-9f2e-4cee-a435-509f70d34bea","Type":"ContainerDied","Data":"dd72a07b6e92f5e48a05eda836050518dc5633a3f19076dc9fd1c0883907787a"}
Nov 03 12:11:47.136569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.136114    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd72a07b6e92f5e48a05eda836050518dc5633a3f19076dc9fd1c0883907787a"
Nov 03 12:11:47.292840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.292794    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:47.468153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.468122    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca\") pod \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") "
Nov 03 12:11:47.468342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.468324    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7\" (UniqueName: \"kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7\") pod \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") "
Nov 03 12:11:47.468415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.468401    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca\") pod \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\" (UID: \"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6\") "
Nov 03 12:11:47.468764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.468724    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca" pod "2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" (UID: "2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:11:47.468920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.468906    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:47.470340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.470316    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca" pod "2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" (UID: "2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:11:47.470501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.470479    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7" pod "2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" (UID: "2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6"). InnerVolumeSpecName "kube-api-access-cbzg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:11:47.570263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.570223    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:47.570263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:47.570265    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbzg7\" (UniqueName: \"kubernetes.io/projected/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6-kube-api-access-cbzg7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:11:48.144198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:48.144167    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"
Nov 03 12:11:48.144599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:48.144194    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj" event={"ID":"2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6","Type":"ContainerDied","Data":"da95953d91dfe455a9cb3c20fcb7dfd5ff8879ad9663c1db74fd589740b75b46"}
Nov 03 12:11:48.144599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:48.144243    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da95953d91dfe455a9cb3c20fcb7dfd5ff8879ad9663c1db74fd589740b75b46"
Nov 03 12:11:59.044187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:59.044150    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"]
Nov 03 12:11:59.090498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:11:59.090463    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eab9b0a-jzvrw"]
Nov 03 12:12:00.190849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.190811    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"]
Nov 03 12:12:00.191576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.191554    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e7196841-9f2e-4cee-a435-509f70d34bea" containerName="revoke-keys"
Nov 03 12:12:00.191671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.191578    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7196841-9f2e-4cee-a435-509f70d34bea" containerName="revoke-keys"
Nov 03 12:12:00.191671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.191626    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" containerName="revoke-keys"
Nov 03 12:12:00.191671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.191635    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" containerName="revoke-keys"
Nov 03 12:12:00.191885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.191791    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7196841-9f2e-4cee-a435-509f70d34bea" containerName="revoke-keys"
Nov 03 12:12:00.191885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.191814    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" containerName="revoke-keys"
Nov 03 12:12:00.196967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.196947    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.245713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.245677    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:12:00.249741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.249712    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"]
Nov 03 12:12:00.290242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.290201    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:12:00.290460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.290307    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:12:00.290722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.290692    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:12:00.324259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.324218    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.324458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.324395    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.324522 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.324469    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqp28\" (UniqueName: \"kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.426775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.426695    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.426985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.426849    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqp28\" (UniqueName: \"kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.426985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.426941    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.429790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.427899    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.442827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.442729    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"]
Nov 03 12:12:00.448135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.448100    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.449024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.449005    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.488926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.488882    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqp28\" (UniqueName: \"kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28\") pod \"maas-api-revoke-sub-5bcfe57f-gjg82\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.522129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.511954    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:00.530485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.530454    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"]
Nov 03 12:12:00.531080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.531002    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.531217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.531196    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.531303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.531287    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8qr\" (UniqueName: \"kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.647147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.647102    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8qr\" (UniqueName: \"kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.647562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.647527    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.647904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.647883    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.650493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.650468    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.658330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.658274    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.683018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.682966    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96729d7c-23f2-4cd2-915c-bdb0440bfaf3" path="/var/lib/kubelet/pods/96729d7c-23f2-4cd2-915c-bdb0440bfaf3/volumes"
Nov 03 12:12:00.704220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.704141    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8qr\" (UniqueName: \"kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr\") pod \"maas-api-revoke-sub-47a3ae1e-jlwk7\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:00.873514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:00.873481    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:01.064854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:01.064683    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"]
Nov 03 12:12:01.073711 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:01.073054    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd83587_b53a_4163_8230_2e560470c41d.slice/crio-8e22229737ce43b2c43f1d2fb7117406294333857f45efca179db1386a03ef27 WatchSource:0}: Error finding container 8e22229737ce43b2c43f1d2fb7117406294333857f45efca179db1386a03ef27: Status 404 returned error can't find the container with id 8e22229737ce43b2c43f1d2fb7117406294333857f45efca179db1386a03ef27
Nov 03 12:12:01.233170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:01.233123    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82" event={"ID":"cdd83587-b53a-4163-8230-2e560470c41d","Type":"ContainerStarted","Data":"8e22229737ce43b2c43f1d2fb7117406294333857f45efca179db1386a03ef27"}
Nov 03 12:12:01.281275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:01.281231    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"]
Nov 03 12:12:01.324011 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:01.313095    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb746f53c_4a17_4ba5_8fdb_7868a9ad9193.slice/crio-48ca3ca29f8fd68a79d0ee6ce4427f7e923c9b1a47bd132dfb270364766ed172 WatchSource:0}: Error finding container 48ca3ca29f8fd68a79d0ee6ce4427f7e923c9b1a47bd132dfb270364766ed172: Status 404 returned error can't find the container with id 48ca3ca29f8fd68a79d0ee6ce4427f7e923c9b1a47bd132dfb270364766ed172
Nov 03 12:12:02.247698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.247459    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82" event={"ID":"cdd83587-b53a-4163-8230-2e560470c41d","Type":"ContainerStarted","Data":"eee62bcf2cd7187ed387a7c3ba56dc054792942435fe8fa7d8b94cdc9c543543"}
Nov 03 12:12:02.249205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.249170    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7" event={"ID":"b746f53c-4a17-4ba5-8fdb-7868a9ad9193","Type":"ContainerStarted","Data":"4e30fc2ace23e5bedc621f93a13956c579b59c7c8a90d09a6ea3013a8b57ccb9"}
Nov 03 12:12:02.249368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.249231    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7" event={"ID":"b746f53c-4a17-4ba5-8fdb-7868a9ad9193","Type":"ContainerStarted","Data":"48ca3ca29f8fd68a79d0ee6ce4427f7e923c9b1a47bd132dfb270364766ed172"}
Nov 03 12:12:02.354270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.354208    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82" podStartSLOduration=2.354188342 podStartE2EDuration="2.354188342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:02.352226895 +0000 UTC m=+1263.923810809" watchObservedRunningTime="2026-11-03 12:12:02.354188342 +0000 UTC m=+1263.925772254"
Nov 03 12:12:02.460483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.460426    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7" podStartSLOduration=2.460408676 podStartE2EDuration="2.460408676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:02.457012696 +0000 UTC m=+1264.028596648" watchObservedRunningTime="2026-11-03 12:12:02.460408676 +0000 UTC m=+1264.031992588"
Nov 03 12:12:02.625828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.625707    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"]
Nov 03 12:12:02.630038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.630018    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.726324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.726221    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"distinct-2-w0-9893f0-kserve-self-signed-certs\""
Nov 03 12:12:02.727941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.727915    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.728087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.728065    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.728222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.728206    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.728274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.728265    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.728317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.728292    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfhb6\" (UniqueName: \"kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.728418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.728396    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.730082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.730061    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"]
Nov 03 12:12:02.829422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829389    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.829609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829459    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.829609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829486    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfhb6\" (UniqueName: \"kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.829609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829511    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.829609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829577    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.829843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829665    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.829942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.829921    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.830609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.830583    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.831042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.831021    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.831926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.831905    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.832312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.832295    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.896327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.896245    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfhb6\" (UniqueName: \"kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6\") pod \"distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:02.947802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:02.947265    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:03.259082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:03.259042    4546 generic.go:356] "Generic (PLEG): container finished" podID="cdd83587-b53a-4163-8230-2e560470c41d" containerID="eee62bcf2cd7187ed387a7c3ba56dc054792942435fe8fa7d8b94cdc9c543543" exitCode=0
Nov 03 12:12:03.259506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:03.259112    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82" event={"ID":"cdd83587-b53a-4163-8230-2e560470c41d","Type":"ContainerDied","Data":"eee62bcf2cd7187ed387a7c3ba56dc054792942435fe8fa7d8b94cdc9c543543"}
Nov 03 12:12:03.261117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:03.261093    4546 generic.go:356] "Generic (PLEG): container finished" podID="b746f53c-4a17-4ba5-8fdb-7868a9ad9193" containerID="4e30fc2ace23e5bedc621f93a13956c579b59c7c8a90d09a6ea3013a8b57ccb9" exitCode=0
Nov 03 12:12:03.261252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:03.261172    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7" event={"ID":"b746f53c-4a17-4ba5-8fdb-7868a9ad9193","Type":"ContainerDied","Data":"4e30fc2ace23e5bedc621f93a13956c579b59c7c8a90d09a6ea3013a8b57ccb9"}
Nov 03 12:12:03.414213 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:03.414181    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod566a9ae7_d9c9_40fb_8436_5f2fa22ecb22.slice/crio-71a2b8a61c37873608e7772c3a034b7b417a7b0c0cc6bb3ce8bb3cfe75bb437e WatchSource:0}: Error finding container 71a2b8a61c37873608e7772c3a034b7b417a7b0c0cc6bb3ce8bb3cfe75bb437e: Status 404 returned error can't find the container with id 71a2b8a61c37873608e7772c3a034b7b417a7b0c0cc6bb3ce8bb3cfe75bb437e
Nov 03 12:12:03.416286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:03.416263    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"]
Nov 03 12:12:04.270082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.270045    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" event={"ID":"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22","Type":"ContainerStarted","Data":"51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf"}
Nov 03 12:12:04.271151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.270813    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" event={"ID":"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22","Type":"ContainerStarted","Data":"71a2b8a61c37873608e7772c3a034b7b417a7b0c0cc6bb3ce8bb3cfe75bb437e"}
Nov 03 12:12:04.384695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.384014    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podStartSLOduration=2.383994128 podStartE2EDuration="2.383994128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:04.380124031 +0000 UTC m=+1265.951707942" watchObservedRunningTime="2026-11-03 12:12:04.383994128 +0000 UTC m=+1265.955578039"
Nov 03 12:12:04.504710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.504665    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:04.554785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.554739    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:04.677498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.677439    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca\") pod \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") "
Nov 03 12:12:04.677714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.677547    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28\" (UniqueName: \"kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28\") pod \"cdd83587-b53a-4163-8230-2e560470c41d\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") "
Nov 03 12:12:04.677714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.677601    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca\") pod \"cdd83587-b53a-4163-8230-2e560470c41d\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") "
Nov 03 12:12:04.677714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.677632    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca\") pod \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") "
Nov 03 12:12:04.677960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.677719    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr\" (UniqueName: \"kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr\") pod \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\" (UID: \"b746f53c-4a17-4ba5-8fdb-7868a9ad9193\") "
Nov 03 12:12:04.677960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.677780    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca\") pod \"cdd83587-b53a-4163-8230-2e560470c41d\" (UID: \"cdd83587-b53a-4163-8230-2e560470c41d\") "
Nov 03 12:12:04.681626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.681591    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca" pod "cdd83587-b53a-4163-8230-2e560470c41d" (UID: "cdd83587-b53a-4163-8230-2e560470c41d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:04.682208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.682182    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca" pod "b746f53c-4a17-4ba5-8fdb-7868a9ad9193" (UID: "b746f53c-4a17-4ba5-8fdb-7868a9ad9193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:04.685507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.685482    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca" pod "cdd83587-b53a-4163-8230-2e560470c41d" (UID: "cdd83587-b53a-4163-8230-2e560470c41d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:04.703588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.703544    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28" pod "cdd83587-b53a-4163-8230-2e560470c41d" (UID: "cdd83587-b53a-4163-8230-2e560470c41d"). InnerVolumeSpecName "kube-api-access-gqp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:04.704803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.704778    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr" pod "b746f53c-4a17-4ba5-8fdb-7868a9ad9193" (UID: "b746f53c-4a17-4ba5-8fdb-7868a9ad9193"). InnerVolumeSpecName "kube-api-access-6s8qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:04.705142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.705122    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca" pod "b746f53c-4a17-4ba5-8fdb-7868a9ad9193" (UID: "b746f53c-4a17-4ba5-8fdb-7868a9ad9193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:04.779428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.779333    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqp28\" (UniqueName: \"kubernetes.io/projected/cdd83587-b53a-4163-8230-2e560470c41d-kube-api-access-gqp28\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:04.779428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.779372    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:04.779428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.779384    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:04.779428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.779394    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8qr\" (UniqueName: \"kubernetes.io/projected/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-kube-api-access-6s8qr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:04.779428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.779403    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd83587-b53a-4163-8230-2e560470c41d-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:04.779428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.779411    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b746f53c-4a17-4ba5-8fdb-7868a9ad9193-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:04.951858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:04.951823    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:05.277687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:05.277658    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"
Nov 03 12:12:05.278200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:05.277702    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82" event={"ID":"cdd83587-b53a-4163-8230-2e560470c41d","Type":"ContainerDied","Data":"8e22229737ce43b2c43f1d2fb7117406294333857f45efca179db1386a03ef27"}
Nov 03 12:12:05.278200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:05.277788    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e22229737ce43b2c43f1d2fb7117406294333857f45efca179db1386a03ef27"
Nov 03 12:12:05.279234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:05.279215    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"
Nov 03 12:12:05.279349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:05.279253    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7" event={"ID":"b746f53c-4a17-4ba5-8fdb-7868a9ad9193","Type":"ContainerDied","Data":"48ca3ca29f8fd68a79d0ee6ce4427f7e923c9b1a47bd132dfb270364766ed172"}
Nov 03 12:12:05.279349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:05.279296    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ca3ca29f8fd68a79d0ee6ce4427f7e923c9b1a47bd132dfb270364766ed172"
Nov 03 12:12:07.650963 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:12:07.650920    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:12:10.898403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.898372    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"]
Nov 03 12:12:10.899152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.899131    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b746f53c-4a17-4ba5-8fdb-7868a9ad9193" containerName="revoke-keys"
Nov 03 12:12:10.899248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.899154    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b746f53c-4a17-4ba5-8fdb-7868a9ad9193" containerName="revoke-keys"
Nov 03 12:12:10.899248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.899170    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cdd83587-b53a-4163-8230-2e560470c41d" containerName="revoke-keys"
Nov 03 12:12:10.899248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.899177    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd83587-b53a-4163-8230-2e560470c41d" containerName="revoke-keys"
Nov 03 12:12:10.899419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.899307    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b746f53c-4a17-4ba5-8fdb-7868a9ad9193" containerName="revoke-keys"
Nov 03 12:12:10.899419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.899323    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd83587-b53a-4163-8230-2e560470c41d" containerName="revoke-keys"
Nov 03 12:12:10.904296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.904275    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:10.947805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.947773    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:10.947929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.947901    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:10.948002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.947987    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvqc\" (UniqueName: \"kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:10.953726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.953708    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:12:10.999813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:10.999784    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:12:11.000058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.000029    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:12:11.000136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.000065    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:12:11.003804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.003779    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"]
Nov 03 12:12:11.048955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.048925    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.049107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.048999    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.049107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.049084    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvqc\" (UniqueName: \"kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.049619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.049595    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.051404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.051383    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.101797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.101765    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvqc\" (UniqueName: \"kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc\") pod \"maas-api-revoke-sub-ebc74b24-r7z24\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.156178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.156084    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"]
Nov 03 12:12:11.160584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.160564    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.214111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.214065    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"]
Nov 03 12:12:11.217467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.217440    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:11.252684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.252643    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.252939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.252815    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrvx\" (UniqueName: \"kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.253027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.253007    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.354021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.353990    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.354211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.354051    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrvx\" (UniqueName: \"kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.354211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.354171    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.355069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.355042    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.356662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.356633    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.406675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.406611    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrvx\" (UniqueName: \"kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx\") pod \"maas-api-revoke-sub-43afc493-jd9ss\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.474381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.474340    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:11.601612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.601338    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"]
Nov 03 12:12:11.601612 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:11.601465    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2942f2_c3a6_4ae0_80c4_9f98f568e77a.slice/crio-55b3e2ba70b31b29632a296aba28ef419e92eae51c8ac03249ad73b7e5ee0b32 WatchSource:0}: Error finding container 55b3e2ba70b31b29632a296aba28ef419e92eae51c8ac03249ad73b7e5ee0b32: Status 404 returned error can't find the container with id 55b3e2ba70b31b29632a296aba28ef419e92eae51c8ac03249ad73b7e5ee0b32
Nov 03 12:12:11.841278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:11.841250    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"]
Nov 03 12:12:11.842396 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:11.842366    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e0fcf2_be21_4033_b808_90af548a0034.slice/crio-405144b8ee72864c952427fd54390df6ffebdfefef40ef31f38e009145f1a264 WatchSource:0}: Error finding container 405144b8ee72864c952427fd54390df6ffebdfefef40ef31f38e009145f1a264: Status 404 returned error can't find the container with id 405144b8ee72864c952427fd54390df6ffebdfefef40ef31f38e009145f1a264
Nov 03 12:12:12.329600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.329560    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24" event={"ID":"af2942f2-c3a6-4ae0-80c4-9f98f568e77a","Type":"ContainerStarted","Data":"79fdf92c7a7ed3e8fb991b127024a1907c47280fd102d6cd67b0e8297270d403"}
Nov 03 12:12:12.330089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.329626    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24" event={"ID":"af2942f2-c3a6-4ae0-80c4-9f98f568e77a","Type":"ContainerStarted","Data":"55b3e2ba70b31b29632a296aba28ef419e92eae51c8ac03249ad73b7e5ee0b32"}
Nov 03 12:12:12.331320 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.331293    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss" event={"ID":"56e0fcf2-be21-4033-b808-90af548a0034","Type":"ContainerStarted","Data":"aaace34c9e82cb01f3f0e129ec41b99e010b759f8f8f1b9a18c806a2174bd673"}
Nov 03 12:12:12.331434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.331336    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss" event={"ID":"56e0fcf2-be21-4033-b808-90af548a0034","Type":"ContainerStarted","Data":"405144b8ee72864c952427fd54390df6ffebdfefef40ef31f38e009145f1a264"}
Nov 03 12:12:12.438112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.438005    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24" podStartSLOduration=2.437990121 podStartE2EDuration="2.437990121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:12.434124886 +0000 UTC m=+1274.005708824" watchObservedRunningTime="2026-11-03 12:12:12.437990121 +0000 UTC m=+1274.009574033"
Nov 03 12:12:12.541620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.541569    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss" podStartSLOduration=1.541554938 podStartE2EDuration="1.541554938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:12.53941432 +0000 UTC m=+1274.110998232" watchObservedRunningTime="2026-11-03 12:12:12.541554938 +0000 UTC m=+1274.113138849"
Nov 03 12:12:12.951853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.951819    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:12.964941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:12.964922    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:13.339215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:13.339125    4546 generic.go:356] "Generic (PLEG): container finished" podID="56e0fcf2-be21-4033-b808-90af548a0034" containerID="aaace34c9e82cb01f3f0e129ec41b99e010b759f8f8f1b9a18c806a2174bd673" exitCode=0
Nov 03 12:12:13.339583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:13.339213    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss" event={"ID":"56e0fcf2-be21-4033-b808-90af548a0034","Type":"ContainerDied","Data":"aaace34c9e82cb01f3f0e129ec41b99e010b759f8f8f1b9a18c806a2174bd673"}
Nov 03 12:12:13.340802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:13.340778    4546 generic.go:356] "Generic (PLEG): container finished" podID="af2942f2-c3a6-4ae0-80c4-9f98f568e77a" containerID="79fdf92c7a7ed3e8fb991b127024a1907c47280fd102d6cd67b0e8297270d403" exitCode=0
Nov 03 12:12:13.340926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:13.340864    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24" event={"ID":"af2942f2-c3a6-4ae0-80c4-9f98f568e77a","Type":"ContainerDied","Data":"79fdf92c7a7ed3e8fb991b127024a1907c47280fd102d6cd67b0e8297270d403"}
Nov 03 12:12:13.352452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:13.352435    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:14.546527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.546498    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:14.586648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.586618    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:14.596179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.596145    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc\" (UniqueName: \"kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc\") pod \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") "
Nov 03 12:12:14.596179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.596184    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca\") pod \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") "
Nov 03 12:12:14.596386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.596209    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca\") pod \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\" (UID: \"af2942f2-c3a6-4ae0-80c4-9f98f568e77a\") "
Nov 03 12:12:14.596606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.596578    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca" pod "af2942f2-c3a6-4ae0-80c4-9f98f568e77a" (UID: "af2942f2-c3a6-4ae0-80c4-9f98f568e77a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:14.596849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.596820    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:14.598541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.598481    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca" pod "af2942f2-c3a6-4ae0-80c4-9f98f568e77a" (UID: "af2942f2-c3a6-4ae0-80c4-9f98f568e77a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:14.598630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.598540    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc" pod "af2942f2-c3a6-4ae0-80c4-9f98f568e77a" (UID: "af2942f2-c3a6-4ae0-80c4-9f98f568e77a"). InnerVolumeSpecName "kube-api-access-tdvqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:14.698157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698109    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx\" (UniqueName: \"kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx\") pod \"56e0fcf2-be21-4033-b808-90af548a0034\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") "
Nov 03 12:12:14.698361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698168    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca\") pod \"56e0fcf2-be21-4033-b808-90af548a0034\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") "
Nov 03 12:12:14.698361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698318    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca\") pod \"56e0fcf2-be21-4033-b808-90af548a0034\" (UID: \"56e0fcf2-be21-4033-b808-90af548a0034\") "
Nov 03 12:12:14.698721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698693    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca" pod "56e0fcf2-be21-4033-b808-90af548a0034" (UID: "56e0fcf2-be21-4033-b808-90af548a0034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:14.698840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698825    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvqc\" (UniqueName: \"kubernetes.io/projected/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-kube-api-access-tdvqc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:14.698969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698845    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2942f2-c3a6-4ae0-80c4-9f98f568e77a-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:14.698969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.698855    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:14.700381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.700356    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx" pod "56e0fcf2-be21-4033-b808-90af548a0034" (UID: "56e0fcf2-be21-4033-b808-90af548a0034"). InnerVolumeSpecName "kube-api-access-bjrvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:14.700469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.700398    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca" pod "56e0fcf2-be21-4033-b808-90af548a0034" (UID: "56e0fcf2-be21-4033-b808-90af548a0034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:14.800115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.800072    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjrvx\" (UniqueName: \"kubernetes.io/projected/56e0fcf2-be21-4033-b808-90af548a0034-kube-api-access-bjrvx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:14.800115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:14.800118    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e0fcf2-be21-4033-b808-90af548a0034-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:15.355170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:15.355144    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"
Nov 03 12:12:15.355371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:15.355183    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss" event={"ID":"56e0fcf2-be21-4033-b808-90af548a0034","Type":"ContainerDied","Data":"405144b8ee72864c952427fd54390df6ffebdfefef40ef31f38e009145f1a264"}
Nov 03 12:12:15.355371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:15.355231    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405144b8ee72864c952427fd54390df6ffebdfefef40ef31f38e009145f1a264"
Nov 03 12:12:15.356554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:15.356536    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"
Nov 03 12:12:15.356686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:15.356552    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24" event={"ID":"af2942f2-c3a6-4ae0-80c4-9f98f568e77a","Type":"ContainerDied","Data":"55b3e2ba70b31b29632a296aba28ef419e92eae51c8ac03249ad73b7e5ee0b32"}
Nov 03 12:12:15.356686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:15.356597    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55b3e2ba70b31b29632a296aba28ef419e92eae51c8ac03249ad73b7e5ee0b32"
Nov 03 12:12:20.348956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.348900    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.496042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.495979    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.543991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.543912    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.566254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.566140    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.608463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.607571    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.748919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.748868    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.792909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:20.792859    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.356920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.356867    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.358452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.358431    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:21.496771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.495665    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.500901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.500861    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:12:21.501474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.501451    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:12:21.553782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.548976    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 12:12:21.553782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.549430    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:12:21.586215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.575853    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.586215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.576300    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:12:21.608620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.607771    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.608620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.608198    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:12:21.867837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.852269    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.933776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.901617    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:12:21.933776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.900939    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Nov 03 12:12:21.933776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:21.908375    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:12:22.318214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:22.318182    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:22.575573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:22.575475    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:22.685227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:22.630919    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:23.077861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:22.631574    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 12:12:23.134268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:22.903459    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:23.136304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:22.910209    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:23.173049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.169780    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:23.173049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.170478    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:12:23.173049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.171009    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:12:23.229775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.225927    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:12:23.340825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.340231    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:12:23.353774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.342516    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:12:23.412804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.411477    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:12:23.457212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.456039    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:12:23.611832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.610689    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:12:23.633813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.628361    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:12:23.633813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:23.628783    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:12:29.043847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:29.043816    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"]
Nov 03 12:12:29.089997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:29.089971    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75e0f11-tpj58"]
Nov 03 12:12:30.388521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.388485    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"]
Nov 03 12:12:30.389089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.389073    4546 cpu_manager.go:414] "containerMap: removing container" podUID="56e0fcf2-be21-4033-b808-90af548a0034" containerName="revoke-keys"
Nov 03 12:12:30.389132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.389090    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e0fcf2-be21-4033-b808-90af548a0034" containerName="revoke-keys"
Nov 03 12:12:30.389132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.389110    4546 cpu_manager.go:414] "containerMap: removing container" podUID="af2942f2-c3a6-4ae0-80c4-9f98f568e77a" containerName="revoke-keys"
Nov 03 12:12:30.389132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.389116    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2942f2-c3a6-4ae0-80c4-9f98f568e77a" containerName="revoke-keys"
Nov 03 12:12:30.389224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.389216    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e0fcf2-be21-4033-b808-90af548a0034" containerName="revoke-keys"
Nov 03 12:12:30.389264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.389225    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2942f2-c3a6-4ae0-80c4-9f98f568e77a" containerName="revoke-keys"
Nov 03 12:12:30.394156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.394135    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.418787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.418727    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.418965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.418949    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qcq\" (UniqueName: \"kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.419064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.419049    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.443803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.443741    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:12:30.489863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.489818    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:12:30.489863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.489833    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:12:30.490078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.489937    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:12:30.493987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.493965    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"]
Nov 03 12:12:30.520040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.520009    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qcq\" (UniqueName: \"kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.520203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.520076    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.520203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.520125    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.520857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.520828    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.522539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.522522    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.573314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.573283    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qcq\" (UniqueName: \"kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq\") pod \"maas-api-revoke-sub-0ba3bf50-sw6rq\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:30.574373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.574343    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1db3853-7a87-4dde-8689-17239ce0d39d" path="/var/lib/kubelet/pods/c1db3853-7a87-4dde-8689-17239ce0d39d/volumes"
Nov 03 12:12:30.707803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:30.707726    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:31.075429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:31.075396    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"]
Nov 03 12:12:31.075540 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:31.075407    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4d1c17_5e62_4362_a4f5_d07efbd33367.slice/crio-9a001284869bb1081bbaa50e3588973da7041adb99f810f82f74b02afd8fffe3 WatchSource:0}: Error finding container 9a001284869bb1081bbaa50e3588973da7041adb99f810f82f74b02afd8fffe3: Status 404 returned error can't find the container with id 9a001284869bb1081bbaa50e3588973da7041adb99f810f82f74b02afd8fffe3
Nov 03 12:12:32.020858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:32.020795    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq" event={"ID":"ef4d1c17-5e62-4362-a4f5-d07efbd33367","Type":"ContainerStarted","Data":"5de66188075c249b6cbf38c160024dbcbfa7e2a2c56ef11a2c84893f317c6ffe"}
Nov 03 12:12:32.021254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:32.020880    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq" event={"ID":"ef4d1c17-5e62-4362-a4f5-d07efbd33367","Type":"ContainerStarted","Data":"9a001284869bb1081bbaa50e3588973da7041adb99f810f82f74b02afd8fffe3"}
Nov 03 12:12:32.126556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:32.126504    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq" podStartSLOduration=2.126487057 podStartE2EDuration="2.126487057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:32.124262128 +0000 UTC m=+1293.695846039" watchObservedRunningTime="2026-11-03 12:12:32.126487057 +0000 UTC m=+1293.698070970"
Nov 03 12:12:33.030321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:33.030283    4546 generic.go:356] "Generic (PLEG): container finished" podID="ef4d1c17-5e62-4362-a4f5-d07efbd33367" containerID="5de66188075c249b6cbf38c160024dbcbfa7e2a2c56ef11a2c84893f317c6ffe" exitCode=0
Nov 03 12:12:33.030709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:33.030356    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq" event={"ID":"ef4d1c17-5e62-4362-a4f5-d07efbd33367","Type":"ContainerDied","Data":"5de66188075c249b6cbf38c160024dbcbfa7e2a2c56ef11a2c84893f317c6ffe"}
Nov 03 12:12:34.195104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.195080    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:34.269072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.269041    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca\") pod \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") "
Nov 03 12:12:34.269248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.269159    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca\") pod \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") "
Nov 03 12:12:34.269248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.269199    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq\" (UniqueName: \"kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq\") pod \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\" (UID: \"ef4d1c17-5e62-4362-a4f5-d07efbd33367\") "
Nov 03 12:12:34.269560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.269536    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca" pod "ef4d1c17-5e62-4362-a4f5-d07efbd33367" (UID: "ef4d1c17-5e62-4362-a4f5-d07efbd33367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:34.269737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.269723    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:34.271366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.271335    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq" pod "ef4d1c17-5e62-4362-a4f5-d07efbd33367" (UID: "ef4d1c17-5e62-4362-a4f5-d07efbd33367"). InnerVolumeSpecName "kube-api-access-j7qcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:34.271477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.271407    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca" pod "ef4d1c17-5e62-4362-a4f5-d07efbd33367" (UID: "ef4d1c17-5e62-4362-a4f5-d07efbd33367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:34.371513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.371427    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qcq\" (UniqueName: \"kubernetes.io/projected/ef4d1c17-5e62-4362-a4f5-d07efbd33367-kube-api-access-j7qcq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:34.371513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:34.371460    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d1c17-5e62-4362-a4f5-d07efbd33367-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:35.044027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:35.043993    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq" event={"ID":"ef4d1c17-5e62-4362-a4f5-d07efbd33367","Type":"ContainerDied","Data":"9a001284869bb1081bbaa50e3588973da7041adb99f810f82f74b02afd8fffe3"}
Nov 03 12:12:35.044027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:35.044013    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"
Nov 03 12:12:35.044285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:35.044039    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a001284869bb1081bbaa50e3588973da7041adb99f810f82f74b02afd8fffe3"
Nov 03 12:12:37.653930 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:12:37.653842    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:12:41.030776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.029958    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.144927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.144871    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.146446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.146324    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.229964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.229900    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.337839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.326165    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.458097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.458041    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.645836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.645542    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.684778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.650048    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:12:41.900830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.900696    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"]
Nov 03 12:12:41.905779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.902072    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ef4d1c17-5e62-4362-a4f5-d07efbd33367" containerName="revoke-keys"
Nov 03 12:12:41.905779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.902104    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4d1c17-5e62-4362-a4f5-d07efbd33367" containerName="revoke-keys"
Nov 03 12:12:41.905779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.902454    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4d1c17-5e62-4362-a4f5-d07efbd33367" containerName="revoke-keys"
Nov 03 12:12:41.925777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.920386    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:41.986783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.982672    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:12:41.986783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.982823    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:12:41.986783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:41.985784    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:12:42.005342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.002172    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.005342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.002280    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.005342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.002427    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fd9z\" (UniqueName: \"kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.005342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.004395    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"]
Nov 03 12:12:42.030780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.024970    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:12:42.041971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.041909    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.043997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.043971    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:12:42.067081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.066898    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.067081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.066978    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.099881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.099842    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.100160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.100131    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.113547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.108159    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.113547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.108252    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.113547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.108353    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fd9z\" (UniqueName: \"kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.113547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.109732    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.130326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.130290    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.162136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.162034    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.162770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.162728    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:12:42.185712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.184344    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.185712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.184814    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:12:42.255076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.255036    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 12:12:42.305478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.305433    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:12:42.357371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.357316    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.384609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.384557    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:42.413862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.412942    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.413862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.413036    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.526061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.490960    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.526061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.498204    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:12:42.711055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.710963    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.731196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.730335    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:12:42.820444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.820410    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fd9z\" (UniqueName: \"kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z\") pod \"maas-api-revoke-sub-0a54034b-wc6ms\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:42.868793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.850501    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:12:42.868793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.859881    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:12:42.919062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:42.919031    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:43.213787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.162813    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.213787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.173877    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:43.213787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.188418    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.339875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.305515    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.365857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.357851    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:43.510023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.504985    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:43.510023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.508909    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.715837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.713724    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.876786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.874908    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.952037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:43.951996    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:44.177706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.177129    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:12:44.177706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.177437    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:44.190661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.188986    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:44.308281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.308232    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:44.430681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.430492    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:12:44.435350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.435307    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:12:44.452571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.441796    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:12:44.452902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.452885    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:12:44.509695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.509509    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:12:44.538816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.510878    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:12:44.546885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.546860    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:12:44.554540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.554508    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:12:44.560797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.560703    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:12:44.811953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.811922    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"]
Nov 03 12:12:44.812510 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:44.812479    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0857f66d_4857_412a_96e9_3edebc47cf2c.slice/crio-338f1345d10098f15a72cac783bbb1530774e1c8336f5469a59eb080388289a8 WatchSource:0}: Error finding container 338f1345d10098f15a72cac783bbb1530774e1c8336f5469a59eb080388289a8: Status 404 returned error can't find the container with id 338f1345d10098f15a72cac783bbb1530774e1c8336f5469a59eb080388289a8
Nov 03 12:12:44.924429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:44.924399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms" event={"ID":"0857f66d-4857-412a-96e9-3edebc47cf2c","Type":"ContainerStarted","Data":"338f1345d10098f15a72cac783bbb1530774e1c8336f5469a59eb080388289a8"}
Nov 03 12:12:45.043730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:45.043696    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"]
Nov 03 12:12:45.047396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:45.047372    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-897a15ef-v5hg4"]
Nov 03 12:12:45.932896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:45.932863    4546 generic.go:356] "Generic (PLEG): container finished" podID="0857f66d-4857-412a-96e9-3edebc47cf2c" containerID="30064496eaa97e5b3a538047ab996a98da4b47cfc80c13528c5e8456f5b86615" exitCode=0
Nov 03 12:12:45.933273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:45.932942    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms" event={"ID":"0857f66d-4857-412a-96e9-3edebc47cf2c","Type":"ContainerDied","Data":"30064496eaa97e5b3a538047ab996a98da4b47cfc80c13528c5e8456f5b86615"}
Nov 03 12:12:46.257266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.257234    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"]
Nov 03 12:12:46.261439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.261419    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.315006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.314974    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"]
Nov 03 12:12:46.326830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.326797    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.326973 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.326873    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfsxx\" (UniqueName: \"kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.327031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.327009    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.428685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.428655    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.428897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.428813    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.428969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.428895    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfsxx\" (UniqueName: \"kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.429571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.429487    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.431500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.431478    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.482556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.482521    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfsxx\" (UniqueName: \"kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx\") pod \"maas-api-revoke-sub-29a51f5c-wtpkf\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.574337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.574263    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d0c509-562a-4a7e-b532-5ddbd5f8b941" path="/var/lib/kubelet/pods/e9d0c509-562a-4a7e-b532-5ddbd5f8b941/volumes"
Nov 03 12:12:46.574701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.574689    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:46.941287 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:46.941239    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38f684d_2c50_4c1f_9d3e_11b1644a7c02.slice/crio-fbf4d37e3a58e6c1f156ddbd8bf5aa827233245fff49cbbcb4df3b679856e945 WatchSource:0}: Error finding container fbf4d37e3a58e6c1f156ddbd8bf5aa827233245fff49cbbcb4df3b679856e945: Status 404 returned error can't find the container with id fbf4d37e3a58e6c1f156ddbd8bf5aa827233245fff49cbbcb4df3b679856e945
Nov 03 12:12:46.941794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:46.941675    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"]
Nov 03 12:12:47.070495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.070471    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:47.138900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.138864    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca\") pod \"0857f66d-4857-412a-96e9-3edebc47cf2c\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") "
Nov 03 12:12:47.139053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.139028    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca\") pod \"0857f66d-4857-412a-96e9-3edebc47cf2c\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") "
Nov 03 12:12:47.139148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.139134    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z\" (UniqueName: \"kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z\") pod \"0857f66d-4857-412a-96e9-3edebc47cf2c\" (UID: \"0857f66d-4857-412a-96e9-3edebc47cf2c\") "
Nov 03 12:12:47.139233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.139208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca" pod "0857f66d-4857-412a-96e9-3edebc47cf2c" (UID: "0857f66d-4857-412a-96e9-3edebc47cf2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:47.139600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.139581    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:47.141359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.141338    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca" pod "0857f66d-4857-412a-96e9-3edebc47cf2c" (UID: "0857f66d-4857-412a-96e9-3edebc47cf2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:47.141445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.141356    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z" pod "0857f66d-4857-412a-96e9-3edebc47cf2c" (UID: "0857f66d-4857-412a-96e9-3edebc47cf2c"). InnerVolumeSpecName "kube-api-access-7fd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:47.240821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.240790    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fd9z\" (UniqueName: \"kubernetes.io/projected/0857f66d-4857-412a-96e9-3edebc47cf2c-kube-api-access-7fd9z\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:47.240821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.240820    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0857f66d-4857-412a-96e9-3edebc47cf2c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:47.949597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.949553    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"
Nov 03 12:12:47.950071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.949546    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms" event={"ID":"0857f66d-4857-412a-96e9-3edebc47cf2c","Type":"ContainerDied","Data":"338f1345d10098f15a72cac783bbb1530774e1c8336f5469a59eb080388289a8"}
Nov 03 12:12:47.950071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.949680    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="338f1345d10098f15a72cac783bbb1530774e1c8336f5469a59eb080388289a8"
Nov 03 12:12:47.951284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.951259    4546 generic.go:356] "Generic (PLEG): container finished" podID="c38f684d-2c50-4c1f-9d3e-11b1644a7c02" containerID="a0d35d96ef27a9f33336166ed2abb3160e38961f2e402088b23efad20287c2a8" exitCode=0
Nov 03 12:12:47.951398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.951351    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf" event={"ID":"c38f684d-2c50-4c1f-9d3e-11b1644a7c02","Type":"ContainerDied","Data":"a0d35d96ef27a9f33336166ed2abb3160e38961f2e402088b23efad20287c2a8"}
Nov 03 12:12:47.951398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:47.951394    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf" event={"ID":"c38f684d-2c50-4c1f-9d3e-11b1644a7c02","Type":"ContainerStarted","Data":"fbf4d37e3a58e6c1f156ddbd8bf5aa827233245fff49cbbcb4df3b679856e945"}
Nov 03 12:12:49.122731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.122709    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:49.267079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.267044    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca\") pod \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") "
Nov 03 12:12:49.267345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.267324    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx\" (UniqueName: \"kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx\") pod \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") "
Nov 03 12:12:49.267426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.267361    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca\") pod \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\" (UID: \"c38f684d-2c50-4c1f-9d3e-11b1644a7c02\") "
Nov 03 12:12:49.267426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.267411    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca" pod "c38f684d-2c50-4c1f-9d3e-11b1644a7c02" (UID: "c38f684d-2c50-4c1f-9d3e-11b1644a7c02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:12:49.267923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.267902    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:49.269799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.269734    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca" pod "c38f684d-2c50-4c1f-9d3e-11b1644a7c02" (UID: "c38f684d-2c50-4c1f-9d3e-11b1644a7c02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:12:49.269799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.269776    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx" pod "c38f684d-2c50-4c1f-9d3e-11b1644a7c02" (UID: "c38f684d-2c50-4c1f-9d3e-11b1644a7c02"). InnerVolumeSpecName "kube-api-access-rfsxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:12:49.368918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.368886    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfsxx\" (UniqueName: \"kubernetes.io/projected/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-kube-api-access-rfsxx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:49.368918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.368922    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38f684d-2c50-4c1f-9d3e-11b1644a7c02-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:12:49.967223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.967188    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf" event={"ID":"c38f684d-2c50-4c1f-9d3e-11b1644a7c02","Type":"ContainerDied","Data":"fbf4d37e3a58e6c1f156ddbd8bf5aa827233245fff49cbbcb4df3b679856e945"}
Nov 03 12:12:49.967223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.967215    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"
Nov 03 12:12:49.967491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:49.967246    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf4d37e3a58e6c1f156ddbd8bf5aa827233245fff49cbbcb4df3b679856e945"
Nov 03 12:12:51.997005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.996968    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"]
Nov 03 12:12:51.997816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.997795    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0857f66d-4857-412a-96e9-3edebc47cf2c" containerName="revoke-keys"
Nov 03 12:12:51.997924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.997817    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0857f66d-4857-412a-96e9-3edebc47cf2c" containerName="revoke-keys"
Nov 03 12:12:51.997924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.997874    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c38f684d-2c50-4c1f-9d3e-11b1644a7c02" containerName="revoke-keys"
Nov 03 12:12:51.997924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.997883    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38f684d-2c50-4c1f-9d3e-11b1644a7c02" containerName="revoke-keys"
Nov 03 12:12:51.998083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.998023    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38f684d-2c50-4c1f-9d3e-11b1644a7c02" containerName="revoke-keys"
Nov 03 12:12:51.998083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:51.998041    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0857f66d-4857-412a-96e9-3edebc47cf2c" containerName="revoke-keys"
Nov 03 12:12:52.001953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.001928    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.085778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.085707    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"]
Nov 03 12:12:52.103491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103456    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.103681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103529    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.103681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103621    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.103681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103669    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.103874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103776    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.103874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103809    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.103979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103920    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.104020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.103998    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.104072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.104043    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.205784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.205701    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.205784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.205792    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.205839    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.205889    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.205933    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206055    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206120    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206149    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206175    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206213    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206190    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206479    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206628    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.206711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.206646    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.208445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.208425    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.208650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.208629    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.258460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.258381    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.258631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.258580    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.315725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.315691    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:52.733224 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:52.733193    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db91edd_3539_4929_aa6e_64b719feef91.slice/crio-ba7bad8b76591c3d6e51d237ad38d06ae1352dc40e6bee80e1fc38c7dd1f39f4 WatchSource:0}: Error finding container ba7bad8b76591c3d6e51d237ad38d06ae1352dc40e6bee80e1fc38c7dd1f39f4: Status 404 returned error can't find the container with id ba7bad8b76591c3d6e51d237ad38d06ae1352dc40e6bee80e1fc38c7dd1f39f4
Nov 03 12:12:52.733616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.733588    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"]
Nov 03 12:12:52.735980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.735952    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:12:52.736038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.736013    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:12:52.736072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.736040    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:12:52.988985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.988892    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" event={"ID":"9db91edd-3539-4929-aa6e-64b719feef91","Type":"ContainerStarted","Data":"8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f"}
Nov 03 12:12:52.988985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:52.988943    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" event={"ID":"9db91edd-3539-4929-aa6e-64b719feef91","Type":"ContainerStarted","Data":"ba7bad8b76591c3d6e51d237ad38d06ae1352dc40e6bee80e1fc38c7dd1f39f4"}
Nov 03 12:12:53.099211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:53.099156    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podStartSLOduration=2.099140521 podStartE2EDuration="2.099140521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:53.096009193 +0000 UTC m=+1314.667593106" watchObservedRunningTime="2026-11-03 12:12:53.099140521 +0000 UTC m=+1314.670724433"
Nov 03 12:12:53.327106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:53.326994    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:12:53.331958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:53.331923    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:53.332486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:53.332452    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:12:54.043931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:54.043896    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"]
Nov 03 12:12:54.047698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:54.047674    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99eff651-55cv8"]
Nov 03 12:12:54.331960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:54.331874    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:54.332443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:54.332409    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:12:54.578820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:54.578776    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677916aa-2437-4d24-99f2-7957c96918e2" path="/var/lib/kubelet/pods/677916aa-2437-4d24-99f2-7957c96918e2/volumes"
Nov 03 12:12:55.128591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.128553    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"]
Nov 03 12:12:55.136997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.136971    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.172275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.172244    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.172539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.172515    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.172717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.172692    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.172879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.172859    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78q25\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.173118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.173094    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.173312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.173297    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.173431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.173417    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.173549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.173535    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.173738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.173720    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.269227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.265611    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-69fd8043-openshift-default-dockercfg-qfwfh\""
Nov 03 12:12:55.293297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.293080    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"]
Nov 03 12:12:55.294001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.293974    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.294262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.294243    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.295628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.295604    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.295807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.295781    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.295954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.295936    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.296149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.296133    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.296387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.296370    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.296530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.296515    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.296699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.296684    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.296862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.296846    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78q25\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.297120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.297095    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.297623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.297333    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.297623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.297565    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.297623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.297595    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.357292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.303682    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.357292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.355470    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:55.357292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.355520    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:12:55.357292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.356831    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.368501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.367427    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.454122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.454050    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78q25\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.459998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.459972    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:55.927777 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:12:55.918848    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fd6e99_9ce1_4b15_a369_48f35701384d.slice/crio-0ae2510103ee3f256ba6481788736190d9f8ea00506e44db297a5545d45e6bd2 WatchSource:0}: Error finding container 0ae2510103ee3f256ba6481788736190d9f8ea00506e44db297a5545d45e6bd2: Status 404 returned error can't find the container with id 0ae2510103ee3f256ba6481788736190d9f8ea00506e44db297a5545d45e6bd2
Nov 03 12:12:55.931852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.931077    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"]
Nov 03 12:12:55.935303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.935260    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:12:55.935406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.935377    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:12:55.935469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:55.935422    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:12:56.042424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.042391    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" event={"ID":"91fd6e99-9ce1-4b15-a369-48f35701384d","Type":"ContainerStarted","Data":"a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"}
Nov 03 12:12:56.042572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.042443    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" event={"ID":"91fd6e99-9ce1-4b15-a369-48f35701384d","Type":"ContainerStarted","Data":"0ae2510103ee3f256ba6481788736190d9f8ea00506e44db297a5545d45e6bd2"}
Nov 03 12:12:56.155697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.155636    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podStartSLOduration=1.155617583 podStartE2EDuration="1.155617583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:56.150624104 +0000 UTC m=+1317.722208016" watchObservedRunningTime="2026-11-03 12:12:56.155617583 +0000 UTC m=+1317.727201496"
Nov 03 12:12:56.321783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.318794    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:56.321783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.318866    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:12:56.461219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.461172    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:56.466478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:56.466451    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:57.049042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:57.049016    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:57.050119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:57.050101    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:12:57.332964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:57.325146    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:57.332964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:57.331844    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:12:58.317892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:58.317853    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:58.318481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:58.318448    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:12:58.839833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:58.839801    4546 scope.go:122] "RemoveContainer" containerID="f95fcd89d7c11b296c7b7e70fc39121e36100b09d8771b4fb7abce46e03209db"
Nov 03 12:12:58.852439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:58.852409    4546 scope.go:122] "RemoveContainer" containerID="8a4babf2944a98c2816a717dc585bf2131c44988958f00ea9c04d4612becfbf8"
Nov 03 12:12:59.316193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:59.316161    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:12:59.316193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:12:59.316215    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 12:13:00.044930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:00.044894    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"]
Nov 03 12:13:00.093643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:00.093607    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c92f0d-gcjd8"]
Nov 03 12:13:00.319944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:00.319863    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:13:00.320348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:00.320319    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:13:00.321045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:00.321026    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 12:13:00.583734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:00.583646    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1b67c5-d3cf-47fa-8730-747df05791db" path="/var/lib/kubelet/pods/fc1b67c5-d3cf-47fa-8730-747df05791db/volumes"
Nov 03 12:13:05.065775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:05.064898    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"]
Nov 03 12:13:05.091428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:05.091397    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31e0199-v2kf8"]
Nov 03 12:13:06.573984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:06.573950    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c4ea78-ec92-4dcd-bcae-458d684d6716" path="/var/lib/kubelet/pods/41c4ea78-ec92-4dcd-bcae-458d684d6716/volumes"
Nov 03 12:13:07.650930 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:13:07.650894    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:13:16.581926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.581893    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"]
Nov 03 12:13:16.586141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.586122    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.681213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.681160    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:13:16.681213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.681193    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:13:16.681213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.681207    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:13:16.681490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.681396    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:13:16.687068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.687041    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"]
Nov 03 12:13:16.733835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.733806    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.734005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.733940    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.734075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.734047    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjs5\" (UniqueName: \"kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.835592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.835506    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.835775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.835616    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.835775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.835683    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjs5\" (UniqueName: \"kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.836154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.836135    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.838119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.838100    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.889284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.889256    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjs5\" (UniqueName: \"kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5\") pod \"maas-api-revoke-sub-8eeae3cb-z7jpg\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:16.900269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:16.900247    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:17.268157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:17.268104    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"]
Nov 03 12:13:17.268443 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:17.268422    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098d9b59_e9bc_44e3_9382_de1f3e4e2ef3.slice/crio-babe42965992438b7a115b91d5ba8164f79df8380526a85a110aa98676135170 WatchSource:0}: Error finding container babe42965992438b7a115b91d5ba8164f79df8380526a85a110aa98676135170: Status 404 returned error can't find the container with id babe42965992438b7a115b91d5ba8164f79df8380526a85a110aa98676135170
Nov 03 12:13:18.211656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:18.211625    4546 generic.go:356] "Generic (PLEG): container finished" podID="098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" containerID="0557233be27c8f274226d825eeff4539028ef8ae74dbe356c956f21705a94e38" exitCode=0
Nov 03 12:13:18.212050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:18.211708    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg" event={"ID":"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3","Type":"ContainerDied","Data":"0557233be27c8f274226d825eeff4539028ef8ae74dbe356c956f21705a94e38"}
Nov 03 12:13:18.212050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:18.211841    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg" event={"ID":"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3","Type":"ContainerStarted","Data":"babe42965992438b7a115b91d5ba8164f79df8380526a85a110aa98676135170"}
Nov 03 12:13:19.374380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.374354    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:19.465851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.465823    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5\" (UniqueName: \"kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5\") pod \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") "
Nov 03 12:13:19.466022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.465921    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca\") pod \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") "
Nov 03 12:13:19.466022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.465986    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca\") pod \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\" (UID: \"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3\") "
Nov 03 12:13:19.466231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.466208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca" pod "098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" (UID: "098d9b59-e9bc-44e3-9382-de1f3e4e2ef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:19.466462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.466446    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:19.468122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.468098    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5" pod "098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" (UID: "098d9b59-e9bc-44e3-9382-de1f3e4e2ef3"). InnerVolumeSpecName "kube-api-access-pdjs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:19.468212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.468098    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca" pod "098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" (UID: "098d9b59-e9bc-44e3-9382-de1f3e4e2ef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:13:19.567211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.567180    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:19.567211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:19.567208    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjs5\" (UniqueName: \"kubernetes.io/projected/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3-kube-api-access-pdjs5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:20.226574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:20.226542    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"
Nov 03 12:13:20.226574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:20.226565    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg" event={"ID":"098d9b59-e9bc-44e3-9382-de1f3e4e2ef3","Type":"ContainerDied","Data":"babe42965992438b7a115b91d5ba8164f79df8380526a85a110aa98676135170"}
Nov 03 12:13:20.226858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:20.226611    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="babe42965992438b7a115b91d5ba8164f79df8380526a85a110aa98676135170"
Nov 03 12:13:22.990013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:22.989985    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"]
Nov 03 12:13:22.990591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:22.990576    4546 cpu_manager.go:414] "containerMap: removing container" podUID="098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" containerName="revoke-keys"
Nov 03 12:13:22.990638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:22.990593    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" containerName="revoke-keys"
Nov 03 12:13:22.990699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:22.990690    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" containerName="revoke-keys"
Nov 03 12:13:22.993826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:22.993808    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.043089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.043057    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:13:23.043415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.043186    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:13:23.090021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.089991    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:13:23.090021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.090014    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:13:23.094858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.094830    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"]
Nov 03 12:13:23.112105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.112077    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.112241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.112195    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kgn\" (UniqueName: \"kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.112331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.112313    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.213678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.213652    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.213872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.213740    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kgn\" (UniqueName: \"kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.213872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.213854    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.214441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.214412    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.216112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.216084    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.267202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.267146    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kgn\" (UniqueName: \"kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn\") pod \"maas-api-revoke-sub-7da65ecf-6gbzn\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.314535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.314513    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:23.690294 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:23.690266    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d1aad2_3cc7_4c05_9bec_7468761ad6c8.slice/crio-caf8f9b8878c5146fb437a052ef76867945f5416aff5cdae5cc432a0c5bc48da WatchSource:0}: Error finding container caf8f9b8878c5146fb437a052ef76867945f5416aff5cdae5cc432a0c5bc48da: Status 404 returned error can't find the container with id caf8f9b8878c5146fb437a052ef76867945f5416aff5cdae5cc432a0c5bc48da
Nov 03 12:13:23.690430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:23.690354    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"]
Nov 03 12:13:24.254859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:24.254822    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn" event={"ID":"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8","Type":"ContainerStarted","Data":"d22446977c0ac9e8bf893d8006d7a7dfa8440ef9df356a9919978db64cdd00d7"}
Nov 03 12:13:24.255299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:24.254886    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn" event={"ID":"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8","Type":"ContainerStarted","Data":"caf8f9b8878c5146fb437a052ef76867945f5416aff5cdae5cc432a0c5bc48da"}
Nov 03 12:13:24.364515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:24.364406    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn" podStartSLOduration=2.364393323 podStartE2EDuration="2.364393323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:24.360311511 +0000 UTC m=+1345.931895415" watchObservedRunningTime="2026-11-03 12:13:24.364393323 +0000 UTC m=+1345.935977234"
Nov 03 12:13:24.785782 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:13:24.785740    4546 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d1aad2_3cc7_4c05_9bec_7468761ad6c8.slice/crio-d22446977c0ac9e8bf893d8006d7a7dfa8440ef9df356a9919978db64cdd00d7.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 12:13:25.263675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:25.263643    4546 generic.go:356] "Generic (PLEG): container finished" podID="e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" containerID="d22446977c0ac9e8bf893d8006d7a7dfa8440ef9df356a9919978db64cdd00d7" exitCode=0
Nov 03 12:13:25.264078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:25.263724    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn" event={"ID":"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8","Type":"ContainerDied","Data":"d22446977c0ac9e8bf893d8006d7a7dfa8440ef9df356a9919978db64cdd00d7"}
Nov 03 12:13:26.414007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.413986    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:26.561502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.561426    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca\") pod \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") "
Nov 03 12:13:26.561502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.561490    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn\" (UniqueName: \"kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn\") pod \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") "
Nov 03 12:13:26.561728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.561547    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca\") pod \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\" (UID: \"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8\") "
Nov 03 12:13:26.561860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.561834    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca" pod "e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" (UID: "e8d1aad2-3cc7-4c05-9bec-7468761ad6c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:26.562211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.562185    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:26.563722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.563697    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn" pod "e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" (UID: "e8d1aad2-3cc7-4c05-9bec-7468761ad6c8"). InnerVolumeSpecName "kube-api-access-24kgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:26.563962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.563943    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca" pod "e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" (UID: "e8d1aad2-3cc7-4c05-9bec-7468761ad6c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:13:26.663335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.663306    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24kgn\" (UniqueName: \"kubernetes.io/projected/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-kube-api-access-24kgn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:26.663335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:26.663336    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:27.279799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.279763    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"]
Nov 03 12:13:27.280613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.280593    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" containerName="revoke-keys"
Nov 03 12:13:27.280613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.280611    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" containerName="revoke-keys"
Nov 03 12:13:27.280841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.280806    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" containerName="revoke-keys"
Nov 03 12:13:27.285585 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.285337    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.287207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.287175    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn" event={"ID":"e8d1aad2-3cc7-4c05-9bec-7468761ad6c8","Type":"ContainerDied","Data":"caf8f9b8878c5146fb437a052ef76867945f5416aff5cdae5cc432a0c5bc48da"}
Nov 03 12:13:27.287331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.287226    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caf8f9b8878c5146fb437a052ef76867945f5416aff5cdae5cc432a0c5bc48da"
Nov 03 12:13:27.287331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.287254    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"
Nov 03 12:13:27.336859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.336830    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"]
Nov 03 12:13:27.383453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.383422    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"]
Nov 03 12:13:27.387783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.387707    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.415884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.415262    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"]
Nov 03 12:13:27.473667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.473631    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.473928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.473693    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmml9\" (UniqueName: \"kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.473928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.473737    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.575481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.575383    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhgb\" (UniqueName: \"kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.575705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.575473    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.575705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.575600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.575991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.575708    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmml9\" (UniqueName: \"kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.575991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.575810    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.575991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.575966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.577080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.577053    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.580352 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.580324    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.629601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.629573    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmml9\" (UniqueName: \"kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9\") pod \"maas-api-revoke-sub-fe07748b-bk5nq\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:27.677345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.677312    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.677529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.677387    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhgb\" (UniqueName: \"kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.677529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.677438    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.678025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.678000    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.679963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.679944    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.729849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.729825    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhgb\" (UniqueName: \"kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb\") pod \"maas-api-revoke-sub-d32f8f70-22n82\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:27.908455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:27.908387    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:28.004210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.004184    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:28.045784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.045700    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"]
Nov 03 12:13:28.051022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.050910    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196661de-7825s"]
Nov 03 12:13:28.279346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.279322    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"]
Nov 03 12:13:28.279457 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:28.279359    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693040f0_dc0f_4165_87cb_ebb6e282b477.slice/crio-37bada4b9fddcc1b7f82d3777aec8fd63e76ff541d9f3084dfe1045674e72863 WatchSource:0}: Error finding container 37bada4b9fddcc1b7f82d3777aec8fd63e76ff541d9f3084dfe1045674e72863: Status 404 returned error can't find the container with id 37bada4b9fddcc1b7f82d3777aec8fd63e76ff541d9f3084dfe1045674e72863
Nov 03 12:13:28.295932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.295905    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq" event={"ID":"693040f0-dc0f-4165-87cb-ebb6e282b477","Type":"ContainerStarted","Data":"37bada4b9fddcc1b7f82d3777aec8fd63e76ff541d9f3084dfe1045674e72863"}
Nov 03 12:13:28.374979 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:28.374949    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a680f5_ed92_4b24_8d1f_354556aa7ad5.slice/crio-c36041ceeaede2a7104570407ef46945e4c7f563733af78c0d91aff11a085da9 WatchSource:0}: Error finding container c36041ceeaede2a7104570407ef46945e4c7f563733af78c0d91aff11a085da9: Status 404 returned error can't find the container with id c36041ceeaede2a7104570407ef46945e4c7f563733af78c0d91aff11a085da9
Nov 03 12:13:28.376140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.376115    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"]
Nov 03 12:13:28.575605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:28.575534    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135b1d9c-b2fa-4838-a176-ff7c840c9080" path="/var/lib/kubelet/pods/135b1d9c-b2fa-4838-a176-ff7c840c9080/volumes"
Nov 03 12:13:29.312317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:29.312282    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82" event={"ID":"01a680f5-ed92-4b24-8d1f-354556aa7ad5","Type":"ContainerStarted","Data":"438a23862f6558b380384554fb03c9da75efac68dd4b76e1e73b792e135c8b98"}
Nov 03 12:13:29.312465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:29.312375    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82" event={"ID":"01a680f5-ed92-4b24-8d1f-354556aa7ad5","Type":"ContainerStarted","Data":"c36041ceeaede2a7104570407ef46945e4c7f563733af78c0d91aff11a085da9"}
Nov 03 12:13:29.314457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:29.314430    4546 generic.go:356] "Generic (PLEG): container finished" podID="693040f0-dc0f-4165-87cb-ebb6e282b477" containerID="e242c5792ad881231276411f4b7b7f8f44c344e0847a392c92abdd4111f6d8e0" exitCode=0
Nov 03 12:13:29.314552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:29.314508    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq" event={"ID":"693040f0-dc0f-4165-87cb-ebb6e282b477","Type":"ContainerDied","Data":"e242c5792ad881231276411f4b7b7f8f44c344e0847a392c92abdd4111f6d8e0"}
Nov 03 12:13:29.418557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:29.418513    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82" podStartSLOduration=2.418501232 podStartE2EDuration="2.418501232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:29.417483445 +0000 UTC m=+1350.989067378" watchObservedRunningTime="2026-11-03 12:13:29.418501232 +0000 UTC m=+1350.990085143"
Nov 03 12:13:30.322952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.322920    4546 generic.go:356] "Generic (PLEG): container finished" podID="01a680f5-ed92-4b24-8d1f-354556aa7ad5" containerID="438a23862f6558b380384554fb03c9da75efac68dd4b76e1e73b792e135c8b98" exitCode=0
Nov 03 12:13:30.323381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.323038    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82" event={"ID":"01a680f5-ed92-4b24-8d1f-354556aa7ad5","Type":"ContainerDied","Data":"438a23862f6558b380384554fb03c9da75efac68dd4b76e1e73b792e135c8b98"}
Nov 03 12:13:30.489091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.489067    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:30.511145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.511122    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca\") pod \"693040f0-dc0f-4165-87cb-ebb6e282b477\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") "
Nov 03 12:13:30.511288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.511192    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca\") pod \"693040f0-dc0f-4165-87cb-ebb6e282b477\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") "
Nov 03 12:13:30.511288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.511248    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9\" (UniqueName: \"kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9\") pod \"693040f0-dc0f-4165-87cb-ebb6e282b477\" (UID: \"693040f0-dc0f-4165-87cb-ebb6e282b477\") "
Nov 03 12:13:30.511639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.511610    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca" pod "693040f0-dc0f-4165-87cb-ebb6e282b477" (UID: "693040f0-dc0f-4165-87cb-ebb6e282b477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:30.511902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.511880    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:30.513386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.513366    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca" pod "693040f0-dc0f-4165-87cb-ebb6e282b477" (UID: "693040f0-dc0f-4165-87cb-ebb6e282b477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:13:30.513802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.513784    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9" pod "693040f0-dc0f-4165-87cb-ebb6e282b477" (UID: "693040f0-dc0f-4165-87cb-ebb6e282b477"). InnerVolumeSpecName "kube-api-access-pmml9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:30.614089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.614054    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693040f0-dc0f-4165-87cb-ebb6e282b477-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:30.614293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:30.614103    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmml9\" (UniqueName: \"kubernetes.io/projected/693040f0-dc0f-4165-87cb-ebb6e282b477-kube-api-access-pmml9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:31.166660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.166630    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"]
Nov 03 12:13:31.167269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.167252    4546 cpu_manager.go:414] "containerMap: removing container" podUID="693040f0-dc0f-4165-87cb-ebb6e282b477" containerName="revoke-keys"
Nov 03 12:13:31.167331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.167272    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693040f0-dc0f-4165-87cb-ebb6e282b477" containerName="revoke-keys"
Nov 03 12:13:31.167422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.167411    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="693040f0-dc0f-4165-87cb-ebb6e282b477" containerName="revoke-keys"
Nov 03 12:13:31.171118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.171099    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.222535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.222133    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.222535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.222321    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.222535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.222359    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2t7j\" (UniqueName: \"kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.224575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.224547    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"]
Nov 03 12:13:31.323386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.323324    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.323900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.323428    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.323900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.323453    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2t7j\" (UniqueName: \"kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.324143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.324118    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.326183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.326160    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.331531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.331514    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"
Nov 03 12:13:31.331631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.331554    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq" event={"ID":"693040f0-dc0f-4165-87cb-ebb6e282b477","Type":"ContainerDied","Data":"37bada4b9fddcc1b7f82d3777aec8fd63e76ff541d9f3084dfe1045674e72863"}
Nov 03 12:13:31.331631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.331594    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bada4b9fddcc1b7f82d3777aec8fd63e76ff541d9f3084dfe1045674e72863"
Nov 03 12:13:31.377121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.377092    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2t7j\" (UniqueName: \"kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j\") pod \"maas-api-revoke-sub-1f5629f5-chh7j\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.484615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.484591    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:31.529438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.529409    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:31.627594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.627563    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca\") pod \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") "
Nov 03 12:13:31.627805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.627618    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca\") pod \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") "
Nov 03 12:13:31.627805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.627641    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb\" (UniqueName: \"kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb\") pod \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\" (UID: \"01a680f5-ed92-4b24-8d1f-354556aa7ad5\") "
Nov 03 12:13:31.627995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.627969    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca" pod "01a680f5-ed92-4b24-8d1f-354556aa7ad5" (UID: "01a680f5-ed92-4b24-8d1f-354556aa7ad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:31.628087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.628065    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:31.629954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.629925    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca" pod "01a680f5-ed92-4b24-8d1f-354556aa7ad5" (UID: "01a680f5-ed92-4b24-8d1f-354556aa7ad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:13:31.630054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.629957    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb" pod "01a680f5-ed92-4b24-8d1f-354556aa7ad5" (UID: "01a680f5-ed92-4b24-8d1f-354556aa7ad5"). InnerVolumeSpecName "kube-api-access-gqhgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:31.728930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.728839    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a680f5-ed92-4b24-8d1f-354556aa7ad5-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:31.728930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.728872    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhgb\" (UniqueName: \"kubernetes.io/projected/01a680f5-ed92-4b24-8d1f-354556aa7ad5-kube-api-access-gqhgb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:31.859176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:31.859147    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"]
Nov 03 12:13:31.859285 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:31.859206    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f85ddc4_d217_40d4_8e4b_5283ccc27808.slice/crio-ac09debc34ab9ba6bb5277d3f4e323d35116c7050d2ec696e76e1909c0096333 WatchSource:0}: Error finding container ac09debc34ab9ba6bb5277d3f4e323d35116c7050d2ec696e76e1909c0096333: Status 404 returned error can't find the container with id ac09debc34ab9ba6bb5277d3f4e323d35116c7050d2ec696e76e1909c0096333
Nov 03 12:13:32.340078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:32.340047    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"
Nov 03 12:13:32.340550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:32.340049    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82" event={"ID":"01a680f5-ed92-4b24-8d1f-354556aa7ad5","Type":"ContainerDied","Data":"c36041ceeaede2a7104570407ef46945e4c7f563733af78c0d91aff11a085da9"}
Nov 03 12:13:32.340550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:32.340199    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c36041ceeaede2a7104570407ef46945e4c7f563733af78c0d91aff11a085da9"
Nov 03 12:13:32.342148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:32.342121    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j" event={"ID":"4f85ddc4-d217-40d4-8e4b-5283ccc27808","Type":"ContainerStarted","Data":"bf9160b9c58e008f8feaf67b2086973790a032d7c84a2c47bd7f62c6123a01de"}
Nov 03 12:13:32.342254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:32.342163    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j" event={"ID":"4f85ddc4-d217-40d4-8e4b-5283ccc27808","Type":"ContainerStarted","Data":"ac09debc34ab9ba6bb5277d3f4e323d35116c7050d2ec696e76e1909c0096333"}
Nov 03 12:13:32.449706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:32.449596    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j" podStartSLOduration=1.449583177 podStartE2EDuration="1.449583177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:32.448543353 +0000 UTC m=+1354.020127266" watchObservedRunningTime="2026-11-03 12:13:32.449583177 +0000 UTC m=+1354.021167087"
Nov 03 12:13:33.351819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:33.351787    4546 generic.go:356] "Generic (PLEG): container finished" podID="4f85ddc4-d217-40d4-8e4b-5283ccc27808" containerID="bf9160b9c58e008f8feaf67b2086973790a032d7c84a2c47bd7f62c6123a01de" exitCode=0
Nov 03 12:13:33.352244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:33.351865    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j" event={"ID":"4f85ddc4-d217-40d4-8e4b-5283ccc27808","Type":"ContainerDied","Data":"bf9160b9c58e008f8feaf67b2086973790a032d7c84a2c47bd7f62c6123a01de"}
Nov 03 12:13:34.523195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.523168    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:34.574618 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.574584    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca\") pod \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") "
Nov 03 12:13:34.574740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.574631    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca\") pod \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") "
Nov 03 12:13:34.574824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.574808    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j\" (UniqueName: \"kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j\") pod \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\" (UID: \"4f85ddc4-d217-40d4-8e4b-5283ccc27808\") "
Nov 03 12:13:34.574909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.574888    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca" pod "4f85ddc4-d217-40d4-8e4b-5283ccc27808" (UID: "4f85ddc4-d217-40d4-8e4b-5283ccc27808"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:34.575210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.575188    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:34.576744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.576722    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca" pod "4f85ddc4-d217-40d4-8e4b-5283ccc27808" (UID: "4f85ddc4-d217-40d4-8e4b-5283ccc27808"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:13:34.576977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.576959    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j" pod "4f85ddc4-d217-40d4-8e4b-5283ccc27808" (UID: "4f85ddc4-d217-40d4-8e4b-5283ccc27808"). InnerVolumeSpecName "kube-api-access-f2t7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:34.676809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.676714    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f85ddc4-d217-40d4-8e4b-5283ccc27808-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:34.676809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:34.676768    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2t7j\" (UniqueName: \"kubernetes.io/projected/4f85ddc4-d217-40d4-8e4b-5283ccc27808-kube-api-access-f2t7j\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:35.367579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:35.367552    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"
Nov 03 12:13:35.367779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:35.367596    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j" event={"ID":"4f85ddc4-d217-40d4-8e4b-5283ccc27808","Type":"ContainerDied","Data":"ac09debc34ab9ba6bb5277d3f4e323d35116c7050d2ec696e76e1909c0096333"}
Nov 03 12:13:35.367779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:35.367651    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac09debc34ab9ba6bb5277d3f4e323d35116c7050d2ec696e76e1909c0096333"
Nov 03 12:13:37.651645 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:13:37.651611    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.420703    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"]
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.421770    4546 cpu_manager.go:414] "containerMap: removing container" podUID="01a680f5-ed92-4b24-8d1f-354556aa7ad5" containerName="revoke-keys"
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.425794    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a680f5-ed92-4b24-8d1f-354556aa7ad5" containerName="revoke-keys"
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.425853    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4f85ddc4-d217-40d4-8e4b-5283ccc27808" containerName="revoke-keys"
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.425861    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f85ddc4-d217-40d4-8e4b-5283ccc27808" containerName="revoke-keys"
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.426077    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a680f5-ed92-4b24-8d1f-354556aa7ad5" containerName="revoke-keys"
Nov 03 12:13:40.439721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.426092    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f85ddc4-d217-40d4-8e4b-5283ccc27808" containerName="revoke-keys"
Nov 03 12:13:40.452786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.447248    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.480901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.480861    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"]
Nov 03 12:13:40.641509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.641468    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.641921 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.641898    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.642089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.642069    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqpp\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.642342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.642324    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.759777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.759328    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.759777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.759489    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.759777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.759527    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqpp\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.759777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.759678    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:40.831776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.831203    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7ab4514d17b9\""
Nov 03 12:13:40.859775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.859639    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-x4r42\""
Nov 03 12:13:40.933866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.933832    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:13:40.934225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.934204    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:13:40.935216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.935196    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:13:40.937077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.937053    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:41.210264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.949764    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:13:41.235863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:40.959226    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:41.236093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.002497    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqpp\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:41.236246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.002529    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"]
Nov 03 12:13:41.246045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.234827    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials\") pod \"payload-processing-external-model-778d4c669c-pf5jh\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") " pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:41.335080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.335050    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.376568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.376122    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:41.387542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.386824    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.409242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.388201    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.431408    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.432108    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.432336    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.432393    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldst\" (UniqueName: \"kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.435031    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.435718    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.435858    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.444091    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.456953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.445920    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:13:41.531869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.531835    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"]
Nov 03 12:13:41.567788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.562052    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.567788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.562143    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.567788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.562210    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldst\" (UniqueName: \"kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.579775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.569010    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.665865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.648448    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.665865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.648547    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.665865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.655396    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldst\" (UniqueName: \"kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst\") pod \"maas-api-revoke-sub-5cc42a51-dzx5v\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:41.683884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.683839    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.894847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.887674    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"]
Nov 03 12:13:41.918821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:41.916067    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:42.061012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.060969    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"]
Nov 03 12:13:42.061384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.061357    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.147829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.122787    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-86d7ad-openshift-default-dockercfg-xd2bl\""
Nov 03 12:13:42.197797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.179984    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.198349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.198314    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.198568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.198547    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.198694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.198678    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.198844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.198825    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.199002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.198985    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqg9c\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.199167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.199150    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.199311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.199295    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.199502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.199483    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.199730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.199709    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.307583    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.307692    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.307814    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.307880    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.307941    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.307969    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.308011    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.308059    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqg9c\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.308119    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.308596    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.308925    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.311047    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.311283    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.320776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.313487    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.347900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.347722    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.359781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.359309    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.395815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.387219    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.395815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.388073    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:13:42.395815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.393553    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.397101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.395941    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:13:42.469798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.436352    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.469798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.436846    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:13:42.482808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.474292    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqg9c\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.538667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.538631    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token\") pod \"e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") " pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:42.590965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.583865    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:42.590965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.583933    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.659780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.651015    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.727893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.727850    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.750735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:42.750442    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:43.267806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.208847    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:13:43.593807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.550008    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.593807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.550825    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:13:43.664467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.595386    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.664467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.636085    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.840776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.825954    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 12:13:43.976986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.952129    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.976986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:43.973676    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.053254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.053203    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Nov 03 12:13:44.244010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.243951    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:13:44.274778 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:13:44.248538    4546 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.638s"
Nov 03 12:13:44.298975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.277124    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": context deadline exceeded"
Nov 03 12:13:44.299482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.299459    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"]
Nov 03 12:13:44.299822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.299805    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:13:44.316409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.316382    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:13:44.317262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.316735    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:13:44.322042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.321873    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:44.322042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.321944    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.590434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.559893    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:13:44.590434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.563841    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.600820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.599849    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.883016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.872882    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.883016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:44.873265    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:13:45.025835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.021697    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.083826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.083196    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.083826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.083561    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:13:45.135548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.133836    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.263981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.262499    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.265491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.265457    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kpcdf" podUID="5b0a55f3-2561-489e-aa3e-ebeb24ca66b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.279603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.279515    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Nov 03 12:13:45.391126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.303091    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.526733 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:45.526701    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5832fc_415a_40c6_a521_33a24588a0fd.slice/crio-cbab7c60acac144dbde2f6e34f0b72158e0b6d34a2e22057772a93fd88b310bc WatchSource:0}: Error finding container cbab7c60acac144dbde2f6e34f0b72158e0b6d34a2e22057772a93fd88b310bc: Status 404 returned error can't find the container with id cbab7c60acac144dbde2f6e34f0b72158e0b6d34a2e22057772a93fd88b310bc
Nov 03 12:13:45.603777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.601738    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.625257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.625220    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.626396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.626357    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 12:13:45.932716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.919156    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh" event={"ID":"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695","Type":"ContainerStarted","Data":"9663d6c2246e4418d44723a039f25859ff1deb37298e7ea77f8d085f9259ee31"}
Nov 03 12:13:45.932716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:45.920724    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.054835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.038909    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.065197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.065148    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c" podUID="bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.089902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.089861    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.164844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.136786    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.292774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.291866    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.436358 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:46.436249    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca94ff8_2b4f_4d80_a494_f6c685a91394.slice/crio-5b24c24e0780008a8dcf4b47340fe3fb2beaf4f8b2edec80850519236e4c47b5 WatchSource:0}: Error finding container 5b24c24e0780008a8dcf4b47340fe3fb2beaf4f8b2edec80850519236e4c47b5: Status 404 returned error can't find the container with id 5b24c24e0780008a8dcf4b47340fe3fb2beaf4f8b2edec80850519236e4c47b5
Nov 03 12:13:46.492474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.492427    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 03 12:13:46.531845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.527804    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:13:46.531845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.527902    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:13:46.531845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.527949    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:13:46.613602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.613566    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:13:46.616910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.616879    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Nov 03 12:13:46.617110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.510546    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:46.620848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.620811    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.906500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.869134    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 12:13:46.906500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:46.869507    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 12:13:47.120161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.120122    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 12:13:47.195541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.192902    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t5gw2" podUID="5a118d42-600f-4c2f-98eb-c6d9ec38776a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:47.195541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.193501    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:13:47.240026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.239994    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:47.240268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.240241    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:47.240736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.240711    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 12:13:47.240850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.240786    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:13:47.276161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.262612    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5cvqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:47.276161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.262662    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:47.278443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.278422    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"]
Nov 03 12:13:47.325141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.325066    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" event={"ID":"bca94ff8-2b4f-4d80-a494-f6c685a91394","Type":"ContainerStarted","Data":"5b24c24e0780008a8dcf4b47340fe3fb2beaf4f8b2edec80850519236e4c47b5"}
Nov 03 12:13:47.342032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.341046    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh" event={"ID":"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695","Type":"ContainerStarted","Data":"0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d"}
Nov 03 12:13:47.342032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.341968    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:47.466958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.466902    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:47.578291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.577596    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:13:47.583440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.583391    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v" event={"ID":"8f5832fc-415a-40c6-a521-33a24588a0fd","Type":"ContainerStarted","Data":"cbab7c60acac144dbde2f6e34f0b72158e0b6d34a2e22057772a93fd88b310bc"}
Nov 03 12:13:47.583938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.583871    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:13:47.584070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.583960    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:13:47.585347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.585328    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 12:13:47.609046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.602962    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:13:47.609046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.606097    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:13:47.679914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:47.679887    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"]
Nov 03 12:13:48.342222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.342161    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh" podStartSLOduration=8.342128909 podStartE2EDuration="8.342128909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:48.340460459 +0000 UTC m=+1369.912044372" watchObservedRunningTime="2026-11-03 12:13:48.342128909 +0000 UTC m=+1369.913712821"
Nov 03 12:13:48.591794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.591729    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" event={"ID":"bca94ff8-2b4f-4d80-a494-f6c685a91394","Type":"ContainerStarted","Data":"59efadd11dc50c08ddfaca9de1da1d48d1464abf5d2e6a9a51ab8b349b99e3f2"}
Nov 03 12:13:48.593376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.593308    4546 generic.go:356] "Generic (PLEG): container finished" podID="8f5832fc-415a-40c6-a521-33a24588a0fd" containerID="a3393b76984f34c016bb6241306edc52861fc23b3cea5c9954ddb622a3415173" exitCode=0
Nov 03 12:13:48.593523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.593389    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v" event={"ID":"8f5832fc-415a-40c6-a521-33a24588a0fd","Type":"ContainerDied","Data":"a3393b76984f34c016bb6241306edc52861fc23b3cea5c9954ddb622a3415173"}
Nov 03 12:13:48.698801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.698720    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podStartSLOduration=7.698705393 podStartE2EDuration="7.698705393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:48.698091024 +0000 UTC m=+1370.269674950" watchObservedRunningTime="2026-11-03 12:13:48.698705393 +0000 UTC m=+1370.270289307"
Nov 03 12:13:48.752565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.752531    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:48.758358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:48.758333    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:49.251128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.251094    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"]
Nov 03 12:13:49.251305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.251289    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" containerID="cri-o://e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa" gracePeriod=30
Nov 03 12:13:49.601742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.601665    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:49.602705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.602685    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:13:49.852292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.852211    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:49.873284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.873257    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca\") pod \"8f5832fc-415a-40c6-a521-33a24588a0fd\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") "
Nov 03 12:13:49.873420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.873333    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca\") pod \"8f5832fc-415a-40c6-a521-33a24588a0fd\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") "
Nov 03 12:13:49.873482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.873466    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst\" (UniqueName: \"kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst\") pod \"8f5832fc-415a-40c6-a521-33a24588a0fd\" (UID: \"8f5832fc-415a-40c6-a521-33a24588a0fd\") "
Nov 03 12:13:49.873766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.873724    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca" pod "8f5832fc-415a-40c6-a521-33a24588a0fd" (UID: "8f5832fc-415a-40c6-a521-33a24588a0fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:49.874097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.874079    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:49.875470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.875448    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca" pod "8f5832fc-415a-40c6-a521-33a24588a0fd" (UID: "8f5832fc-415a-40c6-a521-33a24588a0fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:13:49.875728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.875705    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst" pod "8f5832fc-415a-40c6-a521-33a24588a0fd" (UID: "8f5832fc-415a-40c6-a521-33a24588a0fd"). InnerVolumeSpecName "kube-api-access-lldst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:49.997784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.994873    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5832fc-415a-40c6-a521-33a24588a0fd-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:49.997784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:49.994914    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lldst\" (UniqueName: \"kubernetes.io/projected/8f5832fc-415a-40c6-a521-33a24588a0fd-kube-api-access-lldst\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:50.612016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:50.611984    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"
Nov 03 12:13:50.612016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:50.611991    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v" event={"ID":"8f5832fc-415a-40c6-a521-33a24588a0fd","Type":"ContainerDied","Data":"cbab7c60acac144dbde2f6e34f0b72158e0b6d34a2e22057772a93fd88b310bc"}
Nov 03 12:13:50.612209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:50.612045    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbab7c60acac144dbde2f6e34f0b72158e0b6d34a2e22057772a93fd88b310bc"
Nov 03 12:13:52.903337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:52.903293    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"]
Nov 03 12:13:52.904263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:52.904238    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8f5832fc-415a-40c6-a521-33a24588a0fd" containerName="revoke-keys"
Nov 03 12:13:52.905722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:52.905685    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5832fc-415a-40c6-a521-33a24588a0fd" containerName="revoke-keys"
Nov 03 12:13:52.906027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:52.906000    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5832fc-415a-40c6-a521-33a24588a0fd" containerName="revoke-keys"
Nov 03 12:13:52.912347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:52.912324    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:52.959518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:52.959483    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"]
Nov 03 12:13:53.022773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.015567    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.022773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.015654    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg77f\" (UniqueName: \"kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f\") pod \"payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.022773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.016153    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ac6071\""
Nov 03 12:13:53.022773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.016297    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ac6071-dockercfg-rcjbl\""
Nov 03 12:13:53.096776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.088089    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"]
Nov 03 12:13:53.116775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.105284    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.120924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.119116    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.120924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.119202    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg77f\" (UniqueName: \"kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f\") pod \"payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.121507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.121442    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.135649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.135622    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"]
Nov 03 12:13:53.187321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.187246    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg77f\" (UniqueName: \"kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f\") pod \"payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.220950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.220896    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9dm\" (UniqueName: \"kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm\") pod \"payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.221155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.220975    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.236302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.236263    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:53.322008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.321952    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.322188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.322168    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9dm\" (UniqueName: \"kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm\") pod \"payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.322955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.322929    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.339269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.338912    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:13:53.339269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.338979    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:13:53.388566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.388469    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9dm\" (UniqueName: \"kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm\") pod \"payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.440987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.440872    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:53.662145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.656795    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"]
Nov 03 12:13:53.695150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.695092    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94" event={"ID":"52a978a7-02ca-4a74-896a-899048ec7722","Type":"ContainerStarted","Data":"1291bf87f40748dad8baebb575695ade84db368af541fc760b2eb42b6f10e125"}
Nov 03 12:13:53.716659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.716634    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"]
Nov 03 12:13:53.720358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.720337    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:53.839573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:53.839548    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"]
Nov 03 12:13:53.839884 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:13:53.839853    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a7e4df_942e_4652_82f7_55a399173321.slice/crio-1513a332f2403d61facbd1ae53fd0e8df490636f8d12473be304ba1585cce2ed WatchSource:0}: Error finding container 1513a332f2403d61facbd1ae53fd0e8df490636f8d12473be304ba1585cce2ed: Status 404 returned error can't find the container with id 1513a332f2403d61facbd1ae53fd0e8df490636f8d12473be304ba1585cce2ed
Nov 03 12:13:54.639414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.639385    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:13:54.704284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.704257    4546 generic.go:356] "Generic (PLEG): container finished" podID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerID="a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3" exitCode=0
Nov 03 12:13:54.704459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.704329    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" event={"ID":"91fd6e99-9ce1-4b15-a369-48f35701384d","Type":"ContainerDied","Data":"a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"}
Nov 03 12:13:54.704740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.704721    4546 scope.go:122] "RemoveContainer" containerID="a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"
Nov 03 12:13:54.706245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.706214    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94" event={"ID":"52a978a7-02ca-4a74-896a-899048ec7722","Type":"ContainerStarted","Data":"7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb"}
Nov 03 12:13:54.706366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.706289    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:13:54.706494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.706464    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:13:54.706557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.706542    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:13:54.706601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.706585    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:13:54.708562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.708539    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn" event={"ID":"88a7e4df-942e-4652-82f7-55a399173321","Type":"ContainerStarted","Data":"ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a"}
Nov 03 12:13:54.708652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.708582    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn" event={"ID":"88a7e4df-942e-4652-82f7-55a399173321","Type":"ContainerStarted","Data":"1513a332f2403d61facbd1ae53fd0e8df490636f8d12473be304ba1585cce2ed"}
Nov 03 12:13:54.708652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.708606    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:13:54.710926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.710907    4546 generic.go:356] "Generic (PLEG): container finished" podID="a884db75-8303-42ab-8a47-d94fda05a24c" containerID="e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa" exitCode=0
Nov 03 12:13:54.711017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.710957    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"
Nov 03 12:13:54.711017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.710983    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" event={"ID":"a884db75-8303-42ab-8a47-d94fda05a24c","Type":"ContainerDied","Data":"e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa"}
Nov 03 12:13:54.711100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.711023    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl" event={"ID":"a884db75-8303-42ab-8a47-d94fda05a24c","Type":"ContainerDied","Data":"e8ab89d9a2a63c8f4d5aa48f8d5a479cdbc85eef14f3434a6499330183a31d1b"}
Nov 03 12:13:54.711100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.711043    4546 scope.go:122] "RemoveContainer" containerID="e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa"
Nov 03 12:13:54.720975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.720957    4546 scope.go:122] "RemoveContainer" containerID="e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa"
Nov 03 12:13:54.721257 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:13:54.721237    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa\": container with ID starting with e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa not found: ID does not exist" containerID="e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa"
Nov 03 12:13:54.721332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.721265    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa"} err="failed to get container status \"e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa\": rpc error: code = NotFound desc = could not find container \"e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa\": container with ID starting with e9458ebf7efca38d2d463ce324dbc569b14ac2ade2953541e9ff965c4f1247aa not found: ID does not exist"
Nov 03 12:13:54.772647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.772618    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.772968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.772689    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.772968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.772724    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.772986    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773049    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:54.773154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773073    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773127    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773216    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773267    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773394    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4\") pod \"a884db75-8303-42ab-8a47-d94fda05a24c\" (UID: \"a884db75-8303-42ab-8a47-d94fda05a24c\") "
Nov 03 12:13:54.773440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773416    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:54.774807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773631    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:54.774807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.773731    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:54.774807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.774057    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.774807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.774101    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.775194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.774933    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.775194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.775056    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.775476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.775409    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:54.776203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.776155    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:54.776961 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.776936    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:13:54.777659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.777630    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:54.777942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.777915    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4" pod "a884db75-8303-42ab-8a47-d94fda05a24c" (UID: "a884db75-8303-42ab-8a47-d94fda05a24c"). InnerVolumeSpecName "kube-api-access-xcqn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:54.864784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.864612    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn" podStartSLOduration=1.859823012 podStartE2EDuration="1.859823012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:54.855086981 +0000 UTC m=+1376.426670914" watchObservedRunningTime="2026-11-03 12:13:54.859823012 +0000 UTC m=+1376.431406925"
Nov 03 12:13:54.877116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.877016    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884db75-8303-42ab-8a47-d94fda05a24c-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.877116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.877046    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.877116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.877062    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884db75-8303-42ab-8a47-d94fda05a24c-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.877116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.877080    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcqn4\" (UniqueName: \"kubernetes.io/projected/a884db75-8303-42ab-8a47-d94fda05a24c-kube-api-access-xcqn4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:54.877116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:54.877094    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884db75-8303-42ab-8a47-d94fda05a24c-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:55.119125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:55.118998    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94" podStartSLOduration=3.11898422 podStartE2EDuration="3.11898422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:55.116661735 +0000 UTC m=+1376.688245642" watchObservedRunningTime="2026-11-03 12:13:55.11898422 +0000 UTC m=+1376.690568151"
Nov 03 12:13:55.229482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:55.229408    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"]
Nov 03 12:13:55.239771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:55.239709    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4l4rl"]
Nov 03 12:13:55.745082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:55.745049    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" event={"ID":"91fd6e99-9ce1-4b15-a369-48f35701384d","Type":"ContainerStarted","Data":"f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784"}
Nov 03 12:13:56.461115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:56.461078    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:13:56.462064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:56.462042    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 12:13:56.462163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:56.462089    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 12:13:56.574399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:56.574354    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" path="/var/lib/kubelet/pods/a884db75-8303-42ab-8a47-d94fda05a24c/volumes"
Nov 03 12:13:57.049342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:57.049317    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:13:57.461376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:57.461288    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 12:13:57.461376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:57.461352    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 12:13:58.460574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.460538    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 12:13:58.461011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.460603    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 12:13:58.722830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.721807    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh" podUID="ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" containerName="payload-processing" containerID="cri-o://0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d" gracePeriod=25
Nov 03 12:13:58.890250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.890218    4546 scope.go:122] "RemoveContainer" containerID="d2ebbd0e347b130424ac101c8857d371c19740043158f3a53e992bc651ee3084"
Nov 03 12:13:58.902578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.902552    4546 scope.go:122] "RemoveContainer" containerID="87900464c0de8306e5677c7be14f6c06928f08f2efecccd0c5d1b0d712021198"
Nov 03 12:13:58.940798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.940642    4546 scope.go:122] "RemoveContainer" containerID="6f7d84a5e78c44f79f039ff130357e66d14058b403db2f307979cd746e2f7150"
Nov 03 12:13:58.965744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:58.965714    4546 scope.go:122] "RemoveContainer" containerID="6b4d13408b62cf5a9c7842f433b44c5db458ab9c74c485211ac04f113d56272f"
Nov 03 12:13:59.075553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.075531    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:59.167156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167129    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials\") pod \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") "
Nov 03 12:13:59.167341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167171    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp\") pod \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") "
Nov 03 12:13:59.167341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167204    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume\") pod \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") "
Nov 03 12:13:59.167476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167365    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume\") pod \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\" (UID: \"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695\") "
Nov 03 12:13:59.167691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167658    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume" pod "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" (UID: "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:59.167809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167689    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume" pod "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" (UID: "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:59.168042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167912    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.168042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.167930    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.169361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.169339    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials" pod "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" (UID: "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:59.169654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.169632    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp" pod "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" (UID: "ba9ba14d-aa76-4f83-8d19-cdc0bf88d695"). InnerVolumeSpecName "kube-api-access-nmqpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:59.269506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.269425    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.269506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.269451    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmqpp\" (UniqueName: \"kubernetes.io/projected/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695-kube-api-access-nmqpp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:59.461559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.461521    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 12:13:59.462008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.461580    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 12:13:59.837827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.801215    4546 generic.go:356] "Generic (PLEG): container finished" podID="ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" containerID="0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d" exitCode=0
Nov 03 12:13:59.837827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.801411    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh" event={"ID":"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695","Type":"ContainerDied","Data":"0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d"}
Nov 03 12:13:59.837827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.801469    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh" event={"ID":"ba9ba14d-aa76-4f83-8d19-cdc0bf88d695","Type":"ContainerDied","Data":"9663d6c2246e4418d44723a039f25859ff1deb37298e7ea77f8d085f9259ee31"}
Nov 03 12:13:59.837827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.801511    4546 scope.go:122] "RemoveContainer" containerID="0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d"
Nov 03 12:13:59.837827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.801693    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"
Nov 03 12:13:59.856375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.856358    4546 scope.go:122] "RemoveContainer" containerID="0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d"
Nov 03 12:13:59.856873 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:13:59.856824    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d\": container with ID starting with 0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d not found: ID does not exist" containerID="0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d"
Nov 03 12:13:59.857241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.857212    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d"} err="failed to get container status \"0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d\": rpc error: code = NotFound desc = could not find container \"0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d\": container with ID starting with 0642d986ab3e3e93a18d642cd1a51c53a738b886d9a81549999d98b4ccc4e62d not found: ID does not exist"
Nov 03 12:13:59.979777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.975838    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"]
Nov 03 12:13:59.983326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:13:59.982473    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-778d4c669c-pf5jh"]
Nov 03 12:14:00.046612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.046576    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"]
Nov 03 12:14:00.047301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.047280    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy"
Nov 03 12:14:00.047405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.047306    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy"
Nov 03 12:14:00.047405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.047340    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" containerName="payload-processing"
Nov 03 12:14:00.047405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.047346    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" containerName="payload-processing"
Nov 03 12:14:00.047594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.047447    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a884db75-8303-42ab-8a47-d94fda05a24c" containerName="istio-proxy"
Nov 03 12:14:00.047594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.047458    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" containerName="payload-processing"
Nov 03 12:14:00.052413 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.052388    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.068969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.068945    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"]
Nov 03 12:14:00.101172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.101076    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-86d7ad\""
Nov 03 12:14:00.102083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.101819    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-86d7ad-dockercfg-xgkkj\""
Nov 03 12:14:00.156777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.148815    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.156777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.149016    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546wl\" (UniqueName: \"kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl\") pod \"payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.282890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.282845    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.283089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.283063    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546wl\" (UniqueName: \"kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl\") pod \"payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.284371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.284340    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.342446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.342415    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-546wl\" (UniqueName: \"kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl\") pod \"payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.370506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.370431    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:00.464558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.461410    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 12:14:00.464558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.461472    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 12:14:00.573911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.573881    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9ba14d-aa76-4f83-8d19-cdc0bf88d695" path="/var/lib/kubelet/pods/ba9ba14d-aa76-4f83-8d19-cdc0bf88d695/volumes"
Nov 03 12:14:00.756956 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:14:00.756927    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba5dbef_8bb9_4fad_83d1_ce77a6570985.slice/crio-fd4628b48de4d0175ed36bf84a9441f822c7809db8a5c5cbc22d0df01d87c96f WatchSource:0}: Error finding container fd4628b48de4d0175ed36bf84a9441f822c7809db8a5c5cbc22d0df01d87c96f: Status 404 returned error can't find the container with id fd4628b48de4d0175ed36bf84a9441f822c7809db8a5c5cbc22d0df01d87c96f
Nov 03 12:14:00.757633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.757613    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"]
Nov 03 12:14:00.812633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:00.812598    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj" event={"ID":"1ba5dbef-8bb9-4fad-83d1-ce77a6570985","Type":"ContainerStarted","Data":"fd4628b48de4d0175ed36bf84a9441f822c7809db8a5c5cbc22d0df01d87c96f"}
Nov 03 12:14:01.466123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:01.466094    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:14:01.467268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:01.467243    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:14:01.820805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:01.820689    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj" event={"ID":"1ba5dbef-8bb9-4fad-83d1-ce77a6570985","Type":"ContainerStarted","Data":"f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e"}
Nov 03 12:14:01.820956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:01.820803    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:01.928645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:01.928591    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj" podStartSLOduration=2.928573409 podStartE2EDuration="2.928573409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:14:01.924650452 +0000 UTC m=+1383.496234363" watchObservedRunningTime="2026-11-03 12:14:01.928573409 +0000 UTC m=+1383.500157364"
Nov 03 12:14:05.767101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:05.765924    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:14:05.778482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:05.778447    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:14:06.352007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:06.351958    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.618009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:06.617893    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.618559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:06.618527    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.625811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:06.619174    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.632254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:06.632207    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:07.710724 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:14:07.710530    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:14:12.830005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:12.829979    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:14:37.650873 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:14:37.650841    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:14:43.906269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:43.906237    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"]
Nov 03 12:14:43.910641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:43.910623    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:43.980180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:43.980151    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"]
Nov 03 12:14:44.001845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.001817    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s642n\" (UniqueName: \"kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.001958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.001873    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.002017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.002000    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.002069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.002037    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.002103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.002066    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.002187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.002167    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.005491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.005476    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91cd99\"/\"distinct-w2-91cd99-kserve-self-signed-certs\""
Nov 03 12:14:44.103245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103215    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.103245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103248    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.103493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103281    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.103493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103344    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.103668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103515    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s642n\" (UniqueName: \"kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.103668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103585    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.103812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103774    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.104053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103978    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.104053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.103990    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.106283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.106259    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.106989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.106943    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.158008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.157926    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s642n\" (UniqueName: \"kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n\") pod \"distinct-w2-91cd99-kserve-79d95d548d-jfz52\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") " pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.222680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.222649    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:44.686454 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:14:44.686426    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf7635f_08fc_42d4_b16f_81daee28d04c.slice/crio-bdc73ede38f41d859011ccad3984309d7977890f72da2ac2ffce40ff5d9dde06 WatchSource:0}: Error finding container bdc73ede38f41d859011ccad3984309d7977890f72da2ac2ffce40ff5d9dde06: Status 404 returned error can't find the container with id bdc73ede38f41d859011ccad3984309d7977890f72da2ac2ffce40ff5d9dde06
Nov 03 12:14:44.686602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:44.686474    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"]
Nov 03 12:14:45.545820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:45.545786    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" event={"ID":"5bf7635f-08fc-42d4-b16f-81daee28d04c","Type":"ContainerStarted","Data":"4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c"}
Nov 03 12:14:45.546252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:45.545830    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" event={"ID":"5bf7635f-08fc-42d4-b16f-81daee28d04c","Type":"ContainerStarted","Data":"bdc73ede38f41d859011ccad3984309d7977890f72da2ac2ffce40ff5d9dde06"}
Nov 03 12:14:45.655317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:45.655265    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podStartSLOduration=2.655252195 podStartE2EDuration="2.655252195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:14:45.650741369 +0000 UTC m=+1427.222325280" watchObservedRunningTime="2026-11-03 12:14:45.655252195 +0000 UTC m=+1427.226836105"
Nov 03 12:14:46.225774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:46.222852    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:53.607199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:53.607115    4546 generic.go:356] "Generic (PLEG): container finished" podID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerID="59efadd11dc50c08ddfaca9de1da1d48d1464abf5d2e6a9a51ab8b349b99e3f2" exitCode=0
Nov 03 12:14:53.607630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:53.607200    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" event={"ID":"bca94ff8-2b4f-4d80-a494-f6c685a91394","Type":"ContainerDied","Data":"59efadd11dc50c08ddfaca9de1da1d48d1464abf5d2e6a9a51ab8b349b99e3f2"}
Nov 03 12:14:53.607723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:53.607708    4546 scope.go:122] "RemoveContainer" containerID="59efadd11dc50c08ddfaca9de1da1d48d1464abf5d2e6a9a51ab8b349b99e3f2"
Nov 03 12:14:53.609333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:53.609303    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:14:53.609411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:53.609375    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:14:53.609450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:53.609419    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:14:54.226774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.226675    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:54.242696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.242664    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:54.638709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.638611    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" event={"ID":"bca94ff8-2b4f-4d80-a494-f6c685a91394","Type":"ContainerStarted","Data":"658214e7761a0fada328df211225beb29522795b1001dd0299671ddb277ad40b"}
Nov 03 12:14:54.650427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.650403    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 12:14:54.752926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.752894    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:14:54.753175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.753156    4546 patch_prober.go:29] interesting pod/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 12:14:54.753221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:54.753198    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 12:14:55.752725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:55.752694    4546 patch_prober.go:29] interesting pod/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 12:14:55.753173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:55.752742    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 12:14:56.752541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:56.752507    4546 patch_prober.go:29] interesting pod/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 12:14:56.752713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:56.752573    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 12:14:57.752782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:57.752720    4546 patch_prober.go:29] interesting pod/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 12:14:57.753210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:57.752816    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 12:14:58.753520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:58.753478    4546 patch_prober.go:29] interesting pod/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 12:14:58.753970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:58.753551    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 12:14:59.049863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:59.049782    4546 scope.go:122] "RemoveContainer" containerID="2f6e6add1d41473a801d9daa9e1478467c9ba6b6ab324d0d30bb6d04de3b2189"
Nov 03 12:14:59.060303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:59.060282    4546 scope.go:122] "RemoveContainer" containerID="644489d74c6bda42c8f13d7ccafe1739631e285314df5a4284c6eb39afd87546"
Nov 03 12:14:59.756947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:59.756918    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:14:59.757338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:59.757214    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:14:59.758026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:14:59.757994    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:15:00.164120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.164039    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"]
Nov 03 12:15:00.170356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.170330    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.170623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.170593    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"]
Nov 03 12:15:00.175739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.175717    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"]
Nov 03 12:15:00.175924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.175898    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.181194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.181172    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"]
Nov 03 12:15:00.181311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.181293    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.185464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.185442    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"]
Nov 03 12:15:00.185587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.185553    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.189695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.189674    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"]
Nov 03 12:15:00.189861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.189826    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.193638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.193620    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.219976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.219072    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:15:00.219976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.219072    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:15:00.219976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.219243    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:15:00.219976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.219263    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"]
Nov 03 12:15:00.261918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.261878    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"]
Nov 03 12:15:00.266643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.263059    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.266643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.263722    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhm5\" (UniqueName: \"kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.266643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.263937    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4b5\" (UniqueName: \"kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.266643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.264051    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.269069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.268450    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"]
Nov 03 12:15:00.281907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.275023    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"]
Nov 03 12:15:00.281907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.281737    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"]
Nov 03 12:15:00.286562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.286534    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"]
Nov 03 12:15:00.365124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365094    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7pl\" (UniqueName: \"kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.365325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365149    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4b5\" (UniqueName: \"kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.365325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365266    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f6wk\" (UniqueName: \"kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.365473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365436    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpx7\" (UniqueName: \"kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7\") pod \"maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.365531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365496    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zkd\" (UniqueName: \"kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.365606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365591    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.365655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365620    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.365655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365648    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.365808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365783    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.365885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365825    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.366002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.365973    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.366060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.366033    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhm5\" (UniqueName: \"kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.366450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.366418    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.366650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.366628    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.420158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.420095    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4b5\" (UniqueName: \"kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.420597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.420581    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhm5\" (UniqueName: \"kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.467335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467309    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7pl\" (UniqueName: \"kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.467460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467359    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f6wk\" (UniqueName: \"kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.467460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467406    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpx7\" (UniqueName: \"kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7\") pod \"maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.467576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467461    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zkd\" (UniqueName: \"kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.467576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467495    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.467576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467540    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.467729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467576    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.467729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.467601    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.468231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.468213    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.468304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.468215    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.468339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.468317    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.468402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.468386    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.488441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.488421    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:00.496569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.496549    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:00.522541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.522514    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpx7\" (UniqueName: \"kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7\") pod \"maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.523579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.523551    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f6wk\" (UniqueName: \"kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.523579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.523570    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7pl\" (UniqueName: \"kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.525310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.525284    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zkd\" (UniqueName: \"kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.530993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.530972    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:00.804437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.804406    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:00.812303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.812282    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:00.822456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.822434    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:00.861439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.861414    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"]
Nov 03 12:15:00.873046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.869273    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:00.874632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.874600    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"]
Nov 03 12:15:00.908525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.902788    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:00.908525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.902905    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:00.923453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.920646    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-69fd8043\"/\"openshift-service-ca.crt\""
Nov 03 12:15:00.923453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.920799    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-69fd8043\"/\"e2e-lifecycle-model-69fd8043-kserve-self-signed-certs\""
Nov 03 12:15:00.923453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.920889    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-69fd8043\"/\"default-dockercfg-nqt6j\""
Nov 03 12:15:00.928446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.928033    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-69fd8043\"/\"kube-root-ca.crt\""
Nov 03 12:15:00.936925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.936871    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"]
Nov 03 12:15:00.949551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.946966    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"]
Nov 03 12:15:00.963557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:00.963527    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"]
Nov 03 12:15:00.969881 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:00.969854    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2ae4b8_18bc_48c5_9416_7adacf34cedc.slice/crio-73fa50b54283d890d23432ad4717ad49a79b2f8f4461e79a8616bdd448f44431 WatchSource:0}: Error finding container 73fa50b54283d890d23432ad4717ad49a79b2f8f4461e79a8616bdd448f44431: Status 404 returned error can't find the container with id 73fa50b54283d890d23432ad4717ad49a79b2f8f4461e79a8616bdd448f44431
Nov 03 12:15:01.004934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.004893    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.005099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.005081    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.005219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.005206    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.005515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.005488    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwk6\" (UniqueName: \"kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.005584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.005556    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.005656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.005639    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.005924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.005908    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.007908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.007884    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.107588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.107491    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.107588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.107542    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.107588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.107569    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.107872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.107637    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwk6\" (UniqueName: \"kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.108260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.108133    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.108675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.108638    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.110723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.110701    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.161307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.161276    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwk6\" (UniqueName: \"kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6\") pod \"e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") " pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.190633 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:01.190603    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e01632_9a36_49e9_9aad_cd3f19e995ba.slice/crio-653fe2fa42e08c57cedc3736d49c1a4fea7608600468f041bb13affadb83b7ec WatchSource:0}: Error finding container 653fe2fa42e08c57cedc3736d49c1a4fea7608600468f041bb13affadb83b7ec: Status 404 returned error can't find the container with id 653fe2fa42e08c57cedc3736d49c1a4fea7608600468f041bb13affadb83b7ec
Nov 03 12:15:01.192010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.191986    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"]
Nov 03 12:15:01.208097 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:01.208071    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee12155c_2d7c_47b1_bb02_a9af1ad938ab.slice/crio-555fbc30a17c6ab8d7055977506f680ce05ecef90c5fe7315bad594404551db1 WatchSource:0}: Error finding container 555fbc30a17c6ab8d7055977506f680ce05ecef90c5fe7315bad594404551db1: Status 404 returned error can't find the container with id 555fbc30a17c6ab8d7055977506f680ce05ecef90c5fe7315bad594404551db1
Nov 03 12:15:01.209570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.209535    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"]
Nov 03 12:15:01.215838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.215808    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"]
Nov 03 12:15:01.216012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.215995    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:01.217396 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:01.217370    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb87988_b857_492f_afde_b13d2e744cbb.slice/crio-4e33cb2e6119e31c2bc4adcd5b73679822fe1a40dea0cf722e5556e0154ed9b3 WatchSource:0}: Error finding container 4e33cb2e6119e31c2bc4adcd5b73679822fe1a40dea0cf722e5556e0154ed9b3: Status 404 returned error can't find the container with id 4e33cb2e6119e31c2bc4adcd5b73679822fe1a40dea0cf722e5556e0154ed9b3
Nov 03 12:15:01.694702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.694661    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" event={"ID":"ee12155c-2d7c-47b1-bb02-a9af1ad938ab","Type":"ContainerStarted","Data":"555fbc30a17c6ab8d7055977506f680ce05ecef90c5fe7315bad594404551db1"}
Nov 03 12:15:01.696327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.696298    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" event={"ID":"8e2ae4b8-18bc-48c5-9416-7adacf34cedc","Type":"ContainerStarted","Data":"2286e9adc9f1d1b72fb92ba9c0b14022151b34ed139fecb180d317f3cbb47e3c"}
Nov 03 12:15:01.696488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.696357    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" event={"ID":"8e2ae4b8-18bc-48c5-9416-7adacf34cedc","Type":"ContainerStarted","Data":"73fa50b54283d890d23432ad4717ad49a79b2f8f4461e79a8616bdd448f44431"}
Nov 03 12:15:01.698042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.698017    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerStarted","Data":"a883566df73fbbcd181edee25cbcb56f67f922a8dc2a07dfffa4e01ea19623d7"}
Nov 03 12:15:01.698169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.698058    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerStarted","Data":"653fe2fa42e08c57cedc3736d49c1a4fea7608600468f041bb13affadb83b7ec"}
Nov 03 12:15:01.699658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.699629    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerStarted","Data":"99a5afc87175818a6c877d279ddf9b0f22bf2d8490981e75a1c60f9fe59cba68"}
Nov 03 12:15:01.699770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.699674    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerStarted","Data":"ec86cde9de2dbdc7dabc93a5fe4f2957e1039f3352c427680ecebbd3397d3c71"}
Nov 03 12:15:01.701272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.701245    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerStarted","Data":"ddc5e89f7181cdee45d80756455b519f7e303e65bb6d60d27238c3c32f332dcc"}
Nov 03 12:15:01.701380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.701289    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerStarted","Data":"b8af0261574d4ac4722ef68c40048dd3e0311f7d35f901084db5a62dc24fa476"}
Nov 03 12:15:01.702950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.702930    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerStarted","Data":"e6b9e68d51369495766c878ebd6e5df6c90de7fcb0e7d8ec4ec167edec607968"}
Nov 03 12:15:01.703039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.702966    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerStarted","Data":"4e33cb2e6119e31c2bc4adcd5b73679822fe1a40dea0cf722e5556e0154ed9b3"}
Nov 03 12:15:01.763870 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:01.763838    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74067bff_7993_408c_b816_31ce5428618c.slice/crio-51188bea38449a1647a5cdbc4f4b9893de6f7a4a47eed847791212beff380318 WatchSource:0}: Error finding container 51188bea38449a1647a5cdbc4f4b9893de6f7a4a47eed847791212beff380318: Status 404 returned error can't find the container with id 51188bea38449a1647a5cdbc4f4b9893de6f7a4a47eed847791212beff380318
Nov 03 12:15:01.764847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.764815    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"]
Nov 03 12:15:01.802157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.802118    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" podStartSLOduration=1.802104922 podStartE2EDuration="1.802104922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:01.801527788 +0000 UTC m=+1443.373111700" watchObservedRunningTime="2026-11-03 12:15:01.802104922 +0000 UTC m=+1443.373688832"
Nov 03 12:15:01.909281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:01.909179    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" podStartSLOduration=1.9091620219999998 podStartE2EDuration="1.909162022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:01.906620233 +0000 UTC m=+1443.478204143" watchObservedRunningTime="2026-11-03 12:15:01.909162022 +0000 UTC m=+1443.480745936"
Nov 03 12:15:02.014684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.014634    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" podStartSLOduration=2.01462028 podStartE2EDuration="2.01462028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:02.012061564 +0000 UTC m=+1443.583645474" watchObservedRunningTime="2026-11-03 12:15:02.01462028 +0000 UTC m=+1443.586204242"
Nov 03 12:15:02.120900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.120818    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" podStartSLOduration=2.12079775 podStartE2EDuration="2.12079775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:02.116487435 +0000 UTC m=+1443.688071345" watchObservedRunningTime="2026-11-03 12:15:02.12079775 +0000 UTC m=+1443.692381661"
Nov 03 12:15:02.225821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.225773    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" podStartSLOduration=2.225740545 podStartE2EDuration="2.225740545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:02.221303283 +0000 UTC m=+1443.792887195" watchObservedRunningTime="2026-11-03 12:15:02.225740545 +0000 UTC m=+1443.797324457"
Nov 03 12:15:02.712035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.711936    4546 generic.go:356] "Generic (PLEG): container finished" podID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerID="a883566df73fbbcd181edee25cbcb56f67f922a8dc2a07dfffa4e01ea19623d7" exitCode=60
Nov 03 12:15:02.712190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.712030    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerDied","Data":"a883566df73fbbcd181edee25cbcb56f67f922a8dc2a07dfffa4e01ea19623d7"}
Nov 03 12:15:02.712297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.712280    4546 scope.go:122] "RemoveContainer" containerID="a883566df73fbbcd181edee25cbcb56f67f922a8dc2a07dfffa4e01ea19623d7"
Nov 03 12:15:02.713648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.713630    4546 generic.go:356] "Generic (PLEG): container finished" podID="5d0424de-8707-43e7-a16e-291a166615c7" containerID="99a5afc87175818a6c877d279ddf9b0f22bf2d8490981e75a1c60f9fe59cba68" exitCode=60
Nov 03 12:15:02.713759 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.713696    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerDied","Data":"99a5afc87175818a6c877d279ddf9b0f22bf2d8490981e75a1c60f9fe59cba68"}
Nov 03 12:15:02.713968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.713951    4546 scope.go:122] "RemoveContainer" containerID="99a5afc87175818a6c877d279ddf9b0f22bf2d8490981e75a1c60f9fe59cba68"
Nov 03 12:15:02.715497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.715477    4546 generic.go:356] "Generic (PLEG): container finished" podID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerID="ddc5e89f7181cdee45d80756455b519f7e303e65bb6d60d27238c3c32f332dcc" exitCode=60
Nov 03 12:15:02.715577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.715555    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerDied","Data":"ddc5e89f7181cdee45d80756455b519f7e303e65bb6d60d27238c3c32f332dcc"}
Nov 03 12:15:02.715844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.715739    4546 scope.go:122] "RemoveContainer" containerID="ddc5e89f7181cdee45d80756455b519f7e303e65bb6d60d27238c3c32f332dcc"
Nov 03 12:15:02.717327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.717310    4546 generic.go:356] "Generic (PLEG): container finished" podID="8bb87988-b857-492f-afde-b13d2e744cbb" containerID="e6b9e68d51369495766c878ebd6e5df6c90de7fcb0e7d8ec4ec167edec607968" exitCode=60
Nov 03 12:15:02.717406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.717360    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerDied","Data":"e6b9e68d51369495766c878ebd6e5df6c90de7fcb0e7d8ec4ec167edec607968"}
Nov 03 12:15:02.717637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.717619    4546 scope.go:122] "RemoveContainer" containerID="e6b9e68d51369495766c878ebd6e5df6c90de7fcb0e7d8ec4ec167edec607968"
Nov 03 12:15:02.718998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.718979    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r" event={"ID":"74067bff-7993-408c-b816-31ce5428618c","Type":"ContainerStarted","Data":"b98494b8e8222e7a9e56a067c09d11501639843356ed9a50a59614737b05bfdd"}
Nov 03 12:15:02.719069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.719011    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r" event={"ID":"74067bff-7993-408c-b816-31ce5428618c","Type":"ContainerStarted","Data":"51188bea38449a1647a5cdbc4f4b9893de6f7a4a47eed847791212beff380318"}
Nov 03 12:15:02.720464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.720443    4546 generic.go:356] "Generic (PLEG): container finished" podID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerID="426ec1f1bfc2d6c06dda246f168a0180d2fbfccd91aabe7d79cc56961c7b64a8" exitCode=60
Nov 03 12:15:02.720543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.720517    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" event={"ID":"ee12155c-2d7c-47b1-bb02-a9af1ad938ab","Type":"ContainerDied","Data":"426ec1f1bfc2d6c06dda246f168a0180d2fbfccd91aabe7d79cc56961c7b64a8"}
Nov 03 12:15:02.720705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.720690    4546 scope.go:122] "RemoveContainer" containerID="426ec1f1bfc2d6c06dda246f168a0180d2fbfccd91aabe7d79cc56961c7b64a8"
Nov 03 12:15:02.722040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.722023    4546 generic.go:356] "Generic (PLEG): container finished" podID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerID="2286e9adc9f1d1b72fb92ba9c0b14022151b34ed139fecb180d317f3cbb47e3c" exitCode=60
Nov 03 12:15:02.722124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.722065    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" event={"ID":"8e2ae4b8-18bc-48c5-9416-7adacf34cedc","Type":"ContainerDied","Data":"2286e9adc9f1d1b72fb92ba9c0b14022151b34ed139fecb180d317f3cbb47e3c"}
Nov 03 12:15:02.722242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:02.722230    4546 scope.go:122] "RemoveContainer" containerID="2286e9adc9f1d1b72fb92ba9c0b14022151b34ed139fecb180d317f3cbb47e3c"
Nov 03 12:15:03.216535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.216510    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:03.237031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.236983    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r" podStartSLOduration=3.236971136 podStartE2EDuration="3.236971136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:03.236251914 +0000 UTC m=+1444.807835825" watchObservedRunningTime="2026-11-03 12:15:03.236971136 +0000 UTC m=+1444.808555048"
Nov 03 12:15:03.731798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.731734    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerStarted","Data":"f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea"}
Nov 03 12:15:03.733702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.733679    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerStarted","Data":"57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee"}
Nov 03 12:15:03.735444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.735416    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerStarted","Data":"e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438"}
Nov 03 12:15:03.737272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.737250    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerStarted","Data":"2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c"}
Nov 03 12:15:03.739065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.739043    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" event={"ID":"ee12155c-2d7c-47b1-bb02-a9af1ad938ab","Type":"ContainerStarted","Data":"ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8"}
Nov 03 12:15:03.740653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.740633    4546 generic.go:356] "Generic (PLEG): container finished" podID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerID="b8af3b3b502e96c446a8c8bc3d21b4c9cded66a5b620e89cd47bb07e81ea0924" exitCode=60
Nov 03 12:15:03.740783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.740705    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" event={"ID":"8e2ae4b8-18bc-48c5-9416-7adacf34cedc","Type":"ContainerDied","Data":"b8af3b3b502e96c446a8c8bc3d21b4c9cded66a5b620e89cd47bb07e81ea0924"}
Nov 03 12:15:03.740865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.740802    4546 scope.go:122] "RemoveContainer" containerID="2286e9adc9f1d1b72fb92ba9c0b14022151b34ed139fecb180d317f3cbb47e3c"
Nov 03 12:15:03.741003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:03.740981    4546 scope.go:122] "RemoveContainer" containerID="b8af3b3b502e96c446a8c8bc3d21b4c9cded66a5b620e89cd47bb07e81ea0924"
Nov 03 12:15:03.741283 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:03.741263    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7_odh-ai-gateway-infra(8e2ae4b8-18bc-48c5-9416-7adacf34cedc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc"
Nov 03 12:15:04.363093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.362984    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" podStartSLOduration=4.362969366 podStartE2EDuration="4.362969366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:04.359694721 +0000 UTC m=+1445.931278647" watchObservedRunningTime="2026-11-03 12:15:04.362969366 +0000 UTC m=+1445.934553276"
Nov 03 12:15:04.751340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.751306    4546 generic.go:356] "Generic (PLEG): container finished" podID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerID="ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8" exitCode=60
Nov 03 12:15:04.751510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.751388    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" event={"ID":"ee12155c-2d7c-47b1-bb02-a9af1ad938ab","Type":"ContainerDied","Data":"ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8"}
Nov 03 12:15:04.751510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.751448    4546 scope.go:122] "RemoveContainer" containerID="426ec1f1bfc2d6c06dda246f168a0180d2fbfccd91aabe7d79cc56961c7b64a8"
Nov 03 12:15:04.751791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.751771    4546 scope.go:122] "RemoveContainer" containerID="ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8"
Nov 03 12:15:04.752174 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:04.752154    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869_odh-ai-gateway-infra(ee12155c-2d7c-47b1-bb02-a9af1ad938ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab"
Nov 03 12:15:04.753903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.753884    4546 scope.go:122] "RemoveContainer" containerID="b8af3b3b502e96c446a8c8bc3d21b4c9cded66a5b620e89cd47bb07e81ea0924"
Nov 03 12:15:04.754308 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:04.754287    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7_odh-ai-gateway-infra(8e2ae4b8-18bc-48c5-9416-7adacf34cedc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc"
Nov 03 12:15:04.757381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.756193    4546 generic.go:356] "Generic (PLEG): container finished" podID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerID="f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea" exitCode=60
Nov 03 12:15:04.757381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.756281    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerDied","Data":"f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea"}
Nov 03 12:15:04.757381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.756582    4546 scope.go:122] "RemoveContainer" containerID="f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea"
Nov 03 12:15:04.757381 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:04.757068    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg_odh-ai-gateway-infra(89e01632-9a36-49e9-9aad-cd3f19e995ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba"
Nov 03 12:15:04.763039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.763015    4546 generic.go:356] "Generic (PLEG): container finished" podID="5d0424de-8707-43e7-a16e-291a166615c7" containerID="57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee" exitCode=60
Nov 03 12:15:04.763148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.763094    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerDied","Data":"57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee"}
Nov 03 12:15:04.763361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.763339    4546 scope.go:122] "RemoveContainer" containerID="57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee"
Nov 03 12:15:04.763804 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:04.763728    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r_odh-ai-gateway-infra(5d0424de-8707-43e7-a16e-291a166615c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" podUID="5d0424de-8707-43e7-a16e-291a166615c7"
Nov 03 12:15:04.765494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.765474    4546 generic.go:356] "Generic (PLEG): container finished" podID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerID="e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438" exitCode=60
Nov 03 12:15:04.765593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.765505    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerDied","Data":"e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438"}
Nov 03 12:15:04.765852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.765826    4546 scope.go:122] "RemoveContainer" containerID="e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438"
Nov 03 12:15:04.766214 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:04.766193    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr_odh-ai-gateway-infra(6e59ccbc-3318-44a5-9323-c9a077a213e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2"
Nov 03 12:15:04.767875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.767851    4546 generic.go:356] "Generic (PLEG): container finished" podID="8bb87988-b857-492f-afde-b13d2e744cbb" containerID="2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c" exitCode=60
Nov 03 12:15:04.767966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.767930    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerDied","Data":"2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c"}
Nov 03 12:15:04.768092 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.768077    4546 scope.go:122] "RemoveContainer" containerID="2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c"
Nov 03 12:15:04.768283 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:04.768267    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2_odh-ai-gateway-infra(8bb87988-b857-492f-afde-b13d2e744cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" podUID="8bb87988-b857-492f-afde-b13d2e744cbb"
Nov 03 12:15:04.773344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.773327    4546 scope.go:122] "RemoveContainer" containerID="a883566df73fbbcd181edee25cbcb56f67f922a8dc2a07dfffa4e01ea19623d7"
Nov 03 12:15:04.787176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.787162    4546 scope.go:122] "RemoveContainer" containerID="99a5afc87175818a6c877d279ddf9b0f22bf2d8490981e75a1c60f9fe59cba68"
Nov 03 12:15:04.801781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.801729    4546 scope.go:122] "RemoveContainer" containerID="ddc5e89f7181cdee45d80756455b519f7e303e65bb6d60d27238c3c32f332dcc"
Nov 03 12:15:04.814707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:04.814688    4546 scope.go:122] "RemoveContainer" containerID="e6b9e68d51369495766c878ebd6e5df6c90de7fcb0e7d8ec4ec167edec607968"
Nov 03 12:15:05.777454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:05.777423    4546 scope.go:122] "RemoveContainer" containerID="ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8"
Nov 03 12:15:05.777914 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:05.777801    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869_odh-ai-gateway-infra(ee12155c-2d7c-47b1-bb02-a9af1ad938ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab"
Nov 03 12:15:05.779175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:05.779155    4546 scope.go:122] "RemoveContainer" containerID="f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea"
Nov 03 12:15:05.779487 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:05.779462    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg_odh-ai-gateway-infra(89e01632-9a36-49e9-9aad-cd3f19e995ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba"
Nov 03 12:15:05.780841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:05.780817    4546 scope.go:122] "RemoveContainer" containerID="57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee"
Nov 03 12:15:05.781126 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:05.781109    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r_odh-ai-gateway-infra(5d0424de-8707-43e7-a16e-291a166615c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" podUID="5d0424de-8707-43e7-a16e-291a166615c7"
Nov 03 12:15:05.782383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:05.782354    4546 scope.go:122] "RemoveContainer" containerID="e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438"
Nov 03 12:15:05.782602 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:05.782583    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr_odh-ai-gateway-infra(6e59ccbc-3318-44a5-9323-c9a077a213e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2"
Nov 03 12:15:05.783825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:05.783808    4546 scope.go:122] "RemoveContainer" containerID="2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c"
Nov 03 12:15:05.783999 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:05.783983    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2_odh-ai-gateway-infra(8bb87988-b857-492f-afde-b13d2e744cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" podUID="8bb87988-b857-492f-afde-b13d2e744cbb"
Nov 03 12:15:07.653198 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:07.653161    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:15:11.216741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:11.216707    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:11.229429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:11.229410    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:11.835968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:11.835939    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:15:15.565736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:15.565697    4546 scope.go:122] "RemoveContainer" containerID="ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8"
Nov 03 12:15:16.566078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:16.566052    4546 scope.go:122] "RemoveContainer" containerID="b8af3b3b502e96c446a8c8bc3d21b4c9cded66a5b620e89cd47bb07e81ea0924"
Nov 03 12:15:16.566599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:16.566573    4546 scope.go:122] "RemoveContainer" containerID="f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea"
Nov 03 12:15:16.865555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:16.865522    4546 generic.go:356] "Generic (PLEG): container finished" podID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerID="e2c0e91325ae7760055e9cd58388d1819b6aa89413114221907c98b52f9861d2" exitCode=60
Nov 03 12:15:16.865741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:16.865597    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" event={"ID":"ee12155c-2d7c-47b1-bb02-a9af1ad938ab","Type":"ContainerDied","Data":"e2c0e91325ae7760055e9cd58388d1819b6aa89413114221907c98b52f9861d2"}
Nov 03 12:15:16.865741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:16.865671    4546 scope.go:122] "RemoveContainer" containerID="ab3f0b788fd55bc2c6884e7260631e5cb223ecb8e6e83b5b33d6619bb5c937a8"
Nov 03 12:15:16.866097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:16.866082    4546 scope.go:122] "RemoveContainer" containerID="e2c0e91325ae7760055e9cd58388d1819b6aa89413114221907c98b52f9861d2"
Nov 03 12:15:16.866419 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:16.866399    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869_odh-ai-gateway-infra(ee12155c-2d7c-47b1-bb02-a9af1ad938ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab"
Nov 03 12:15:17.565944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.565920    4546 scope.go:122] "RemoveContainer" containerID="2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c"
Nov 03 12:15:17.566131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.566106    4546 scope.go:122] "RemoveContainer" containerID="e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438"
Nov 03 12:15:17.876713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.876683    4546 generic.go:356] "Generic (PLEG): container finished" podID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerID="a54bd0076e45709c957497213ba9f001a3841b9105d98294bbfaa13de137a4da" exitCode=60
Nov 03 12:15:17.876935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.876765    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" event={"ID":"8e2ae4b8-18bc-48c5-9416-7adacf34cedc","Type":"ContainerDied","Data":"a54bd0076e45709c957497213ba9f001a3841b9105d98294bbfaa13de137a4da"}
Nov 03 12:15:17.876935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.876821    4546 scope.go:122] "RemoveContainer" containerID="b8af3b3b502e96c446a8c8bc3d21b4c9cded66a5b620e89cd47bb07e81ea0924"
Nov 03 12:15:17.877258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.877240    4546 scope.go:122] "RemoveContainer" containerID="a54bd0076e45709c957497213ba9f001a3841b9105d98294bbfaa13de137a4da"
Nov 03 12:15:17.877625 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:17.877599    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7_odh-ai-gateway-infra(8e2ae4b8-18bc-48c5-9416-7adacf34cedc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc"
Nov 03 12:15:17.879852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.879830    4546 generic.go:356] "Generic (PLEG): container finished" podID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerID="0cf030f9c3c1ed20d1af4c1c575e70f0e8b4fbf3418a97fe82a8dca859a64477" exitCode=60
Nov 03 12:15:17.879952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.879862    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerDied","Data":"0cf030f9c3c1ed20d1af4c1c575e70f0e8b4fbf3418a97fe82a8dca859a64477"}
Nov 03 12:15:17.880222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.880206    4546 scope.go:122] "RemoveContainer" containerID="0cf030f9c3c1ed20d1af4c1c575e70f0e8b4fbf3418a97fe82a8dca859a64477"
Nov 03 12:15:17.880485 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:17.880466    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg_odh-ai-gateway-infra(89e01632-9a36-49e9-9aad-cd3f19e995ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba"
Nov 03 12:15:17.891300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.891278    4546 scope.go:122] "RemoveContainer" containerID="f8d44591f7663c3b7977246ee237dc625f1617e3d2c5a2050973fe559da821ea"
Nov 03 12:15:17.984504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:17.984469    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"]
Nov 03 12:15:18.167516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.167492    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:18.268611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.268578    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd\" (UniqueName: \"kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd\") pod \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") "
Nov 03 12:15:18.268847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.268724    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca\") pod \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\" (UID: \"ee12155c-2d7c-47b1-bb02-a9af1ad938ab\") "
Nov 03 12:15:18.269155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.269130    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca" pod "ee12155c-2d7c-47b1-bb02-a9af1ad938ab" (UID: "ee12155c-2d7c-47b1-bb02-a9af1ad938ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:18.269397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.269382    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:18.270967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.270940    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd" pod "ee12155c-2d7c-47b1-bb02-a9af1ad938ab" (UID: "ee12155c-2d7c-47b1-bb02-a9af1ad938ab"). InnerVolumeSpecName "kube-api-access-f8zkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:18.370735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.370702    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zkd\" (UniqueName: \"kubernetes.io/projected/ee12155c-2d7c-47b1-bb02-a9af1ad938ab-kube-api-access-f8zkd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:18.574881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.574849    4546 scope.go:122] "RemoveContainer" containerID="57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee"
Nov 03 12:15:18.892897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.892842    4546 generic.go:356] "Generic (PLEG): container finished" podID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerID="7c479c59a4024c7415511fdd2b4c2f68f4f1b4a34919d24cc75fd59706fbc3fd" exitCode=60
Nov 03 12:15:18.893054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.892922    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerDied","Data":"7c479c59a4024c7415511fdd2b4c2f68f4f1b4a34919d24cc75fd59706fbc3fd"}
Nov 03 12:15:18.893054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.892984    4546 scope.go:122] "RemoveContainer" containerID="e3ee8c92edf79703ab5d150729f5213bda662d88d0e667e7f431e99885a9c438"
Nov 03 12:15:18.893425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.893405    4546 scope.go:122] "RemoveContainer" containerID="7c479c59a4024c7415511fdd2b4c2f68f4f1b4a34919d24cc75fd59706fbc3fd"
Nov 03 12:15:18.893808 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:18.893781    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr_odh-ai-gateway-infra(6e59ccbc-3318-44a5-9323-c9a077a213e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2"
Nov 03 12:15:18.894989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.894970    4546 generic.go:356] "Generic (PLEG): container finished" podID="8bb87988-b857-492f-afde-b13d2e744cbb" containerID="a44b9a41f2b0b4d1486fb353b438b03184a99d3c1995105474ffded553a8e407" exitCode=60
Nov 03 12:15:18.895065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.895037    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerDied","Data":"a44b9a41f2b0b4d1486fb353b438b03184a99d3c1995105474ffded553a8e407"}
Nov 03 12:15:18.895367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.895354    4546 scope.go:122] "RemoveContainer" containerID="a44b9a41f2b0b4d1486fb353b438b03184a99d3c1995105474ffded553a8e407"
Nov 03 12:15:18.895659 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:18.895639    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2_odh-ai-gateway-infra(8bb87988-b857-492f-afde-b13d2e744cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" podUID="8bb87988-b857-492f-afde-b13d2e744cbb"
Nov 03 12:15:18.896410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.896386    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869" event={"ID":"ee12155c-2d7c-47b1-bb02-a9af1ad938ab","Type":"ContainerDied","Data":"555fbc30a17c6ab8d7055977506f680ce05ecef90c5fe7315bad594404551db1"}
Nov 03 12:15:18.896501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.896425    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"
Nov 03 12:15:18.907102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.907085    4546 scope.go:122] "RemoveContainer" containerID="2c1f72f4e68328af9babac3060d2fef90f7b8a39d6fe87a1c966d24569278e6c"
Nov 03 12:15:18.921373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.921354    4546 scope.go:122] "RemoveContainer" containerID="e2c0e91325ae7760055e9cd58388d1819b6aa89413114221907c98b52f9861d2"
Nov 03 12:15:18.997159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:18.997132    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"]
Nov 03 12:15:19.139822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.138823    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"]
Nov 03 12:15:19.154808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.154515    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"]
Nov 03 12:15:19.192171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.190138    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895135-v8869"]
Nov 03 12:15:19.345946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.345911    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:19.381811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.381781    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"]
Nov 03 12:15:19.382653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382628    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.382653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382650    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382697    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382704    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382720    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382727    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382739    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382761    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382802    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382809    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.383232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382966    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.383232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382982    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.383232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.382992    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" containerName="cleanup"
Nov 03 12:15:19.383232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.383006    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.383232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.383016    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.383232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.383027    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.386905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.386881    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.491770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.442164    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-913a17-openshift-default-dockercfg-6fbkb\""
Nov 03 12:15:19.491770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.443674    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:19.491770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.445789    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"]
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.497231    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7\" (UniqueName: \"kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7\") pod \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") "
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.497592    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca\") pod \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\" (UID: \"8e2ae4b8-18bc-48c5-9416-7adacf34cedc\") "
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498445    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca" pod "8e2ae4b8-18bc-48c5-9416-7adacf34cedc" (UID: "8e2ae4b8-18bc-48c5-9416-7adacf34cedc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498539    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498664    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498723    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498790    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498935    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsvd\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.498973    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.499005    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.499060    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.499095    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.510775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.499167    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:19.572082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.558978    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7" pod "8e2ae4b8-18bc-48c5-9416-7adacf34cedc" (UID: "8e2ae4b8-18bc-48c5-9416-7adacf34cedc"). InnerVolumeSpecName "kube-api-access-vtpx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:19.623113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623080    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk\" (UniqueName: \"kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk\") pod \"89e01632-9a36-49e9-9aad-cd3f19e995ba\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") "
Nov 03 12:15:19.623543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623197    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca\") pod \"89e01632-9a36-49e9-9aad-cd3f19e995ba\" (UID: \"89e01632-9a36-49e9-9aad-cd3f19e995ba\") "
Nov 03 12:15:19.623543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623369    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623530    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsvd\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623569    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623644    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623695    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623828    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.623949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.623940    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.624062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.624012    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.624118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.624089    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtpx7\" (UniqueName: \"kubernetes.io/projected/8e2ae4b8-18bc-48c5-9416-7adacf34cedc-kube-api-access-vtpx7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:19.624494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.624467    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.626718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.626693    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.627070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.627047    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.627569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.627543    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk" pod "89e01632-9a36-49e9-9aad-cd3f19e995ba" (UID: "89e01632-9a36-49e9-9aad-cd3f19e995ba"). InnerVolumeSpecName "kube-api-access-8f6wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:19.634103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.628920    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.637125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.637102    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.640515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.640495    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.646147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.644981    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca" pod "89e01632-9a36-49e9-9aad-cd3f19e995ba" (UID: "89e01632-9a36-49e9-9aad-cd3f19e995ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:19.663791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.653197    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.682082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.682027    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsvd\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.692499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.692468    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token\") pod \"e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") " pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.704241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.704219    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:19.727257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.726922    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f6wk\" (UniqueName: \"kubernetes.io/projected/89e01632-9a36-49e9-9aad-cd3f19e995ba-kube-api-access-8f6wk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:19.727257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.726952    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89e01632-9a36-49e9-9aad-cd3f19e995ba-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:19.906680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.906603    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"
Nov 03 12:15:19.906680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.906613    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg" event={"ID":"89e01632-9a36-49e9-9aad-cd3f19e995ba","Type":"ContainerDied","Data":"653fe2fa42e08c57cedc3736d49c1a4fea7608600468f041bb13affadb83b7ec"}
Nov 03 12:15:19.906915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.906681    4546 scope.go:122] "RemoveContainer" containerID="0cf030f9c3c1ed20d1af4c1c575e70f0e8b4fbf3418a97fe82a8dca859a64477"
Nov 03 12:15:19.909518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.909489    4546 generic.go:356] "Generic (PLEG): container finished" podID="5d0424de-8707-43e7-a16e-291a166615c7" containerID="51de73bec4f3b351f006da299b097a09be54bd7da9d4329b682da5968200f67e" exitCode=60
Nov 03 12:15:19.909917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.909566    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerDied","Data":"51de73bec4f3b351f006da299b097a09be54bd7da9d4329b682da5968200f67e"}
Nov 03 12:15:19.911474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.911396    4546 scope.go:122] "RemoveContainer" containerID="51de73bec4f3b351f006da299b097a09be54bd7da9d4329b682da5968200f67e"
Nov 03 12:15:19.911819 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:19.911796    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r_odh-ai-gateway-infra(5d0424de-8707-43e7-a16e-291a166615c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" podUID="5d0424de-8707-43e7-a16e-291a166615c7"
Nov 03 12:15:19.920528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.920504    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"
Nov 03 12:15:19.920642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.920500    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7" event={"ID":"8e2ae4b8-18bc-48c5-9416-7adacf34cedc","Type":"ContainerDied","Data":"73fa50b54283d890d23432ad4717ad49a79b2f8f4461e79a8616bdd448f44431"}
Nov 03 12:15:19.924637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.924134    4546 scope.go:122] "RemoveContainer" containerID="57aa9d9d75f22c20b5a58bca70265e48f1c10640de1eb3893f15af1c590877ee"
Nov 03 12:15:19.926317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.926292    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"]
Nov 03 12:15:19.927822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.927802    4546 cpu_manager.go:414] "containerMap: removing container" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:19.927914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.927823    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:19.927914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.927884    4546 cpu_manager.go:414] "containerMap: removing container" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:19.927914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.927892    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:19.928047 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.927970    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.928047 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.927978    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" containerName="cleanup"
Nov 03 12:15:19.928398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.928380    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:19.928451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.928413    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:19.935818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.935779    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:19.952417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.951682    4546 scope.go:122] "RemoveContainer" containerID="a54bd0076e45709c957497213ba9f001a3841b9105d98294bbfaa13de137a4da"
Nov 03 12:15:19.985339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.983929    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"]
Nov 03 12:15:20.003133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:19.997660    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91cd99-openshift-default-dockercfg-pj65x\""
Nov 03 12:15:20.018389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.018353    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"]
Nov 03 12:15:20.130805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.130040    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"]
Nov 03 12:15:20.131687 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:20.131647    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda995bae5_79c9_4200_921e_3940efb438d0.slice/crio-351af2ec18fe5eb770bc00c885118b1784a856ff99266bb5e91914bb5c5bfcb0 WatchSource:0}: Error finding container 351af2ec18fe5eb770bc00c885118b1784a856ff99266bb5e91914bb5c5bfcb0: Status 404 returned error can't find the container with id 351af2ec18fe5eb770bc00c885118b1784a856ff99266bb5e91914bb5c5bfcb0
Nov 03 12:15:20.135764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.135715    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:15:20.135951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.135926    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:15:20.136110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.136087    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:15:20.144034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144006    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144059    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144215    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144288    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144315    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144339    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzdw\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144430    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144464    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.144514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.144498    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.181936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.181903    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"]
Nov 03 12:15:20.187928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.187904    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-69fd8043-29895135-wtjq7"]
Nov 03 12:15:20.194159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.194140    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:20.245648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.245616    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.245827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.245671    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.245913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.245870    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.245992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.245955    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.245996    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246020    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzdw\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246109    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246114    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246135    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246148    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246208    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246480    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246702    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.246938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.246826    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.248972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.248949    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.249079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.249044    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.253184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.253160    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"]
Nov 03 12:15:20.279851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.279821    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"]
Nov 03 12:15:20.283650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.283623    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895135-k6mpg"]
Nov 03 12:15:20.299407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.299382    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzdw\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.299543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.299454    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.347865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.347833    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5\" (UniqueName: \"kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5\") pod \"6e59ccbc-3318-44a5-9323-c9a077a213e2\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") "
Nov 03 12:15:20.348040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.347951    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca\") pod \"6e59ccbc-3318-44a5-9323-c9a077a213e2\" (UID: \"6e59ccbc-3318-44a5-9323-c9a077a213e2\") "
Nov 03 12:15:20.348412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.348301    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca" pod "6e59ccbc-3318-44a5-9323-c9a077a213e2" (UID: "6e59ccbc-3318-44a5-9323-c9a077a213e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:20.348628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.348610    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e59ccbc-3318-44a5-9323-c9a077a213e2-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:20.350181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.350158    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5" pod "6e59ccbc-3318-44a5-9323-c9a077a213e2" (UID: "6e59ccbc-3318-44a5-9323-c9a077a213e2"). InnerVolumeSpecName "kube-api-access-4bhm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:20.407203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.407179    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:20.449648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.449576    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl\" (UniqueName: \"kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl\") pod \"8bb87988-b857-492f-afde-b13d2e744cbb\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") "
Nov 03 12:15:20.449648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.449634    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca\") pod \"8bb87988-b857-492f-afde-b13d2e744cbb\" (UID: \"8bb87988-b857-492f-afde-b13d2e744cbb\") "
Nov 03 12:15:20.450085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.450060    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca" pod "8bb87988-b857-492f-afde-b13d2e744cbb" (UID: "8bb87988-b857-492f-afde-b13d2e744cbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:20.450218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.450202    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bhm5\" (UniqueName: \"kubernetes.io/projected/6e59ccbc-3318-44a5-9323-c9a077a213e2-kube-api-access-4bhm5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:20.450442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.450224    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb87988-b857-492f-afde-b13d2e744cbb-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:20.451819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.451796    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl" pod "8bb87988-b857-492f-afde-b13d2e744cbb" (UID: "8bb87988-b857-492f-afde-b13d2e744cbb"). InnerVolumeSpecName "kube-api-access-nw7pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:20.552023 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.551995    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7pl\" (UniqueName: \"kubernetes.io/projected/8bb87988-b857-492f-afde-b13d2e744cbb-kube-api-access-nw7pl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:20.559123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.559100    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:20.575044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.575023    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" path="/var/lib/kubelet/pods/89e01632-9a36-49e9-9aad-cd3f19e995ba/volumes"
Nov 03 12:15:20.575560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.575490    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2ae4b8-18bc-48c5-9416-7adacf34cedc" path="/var/lib/kubelet/pods/8e2ae4b8-18bc-48c5-9416-7adacf34cedc/volumes"
Nov 03 12:15:20.576009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.575985    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee12155c-2d7c-47b1-bb02-a9af1ad938ab" path="/var/lib/kubelet/pods/ee12155c-2d7c-47b1-bb02-a9af1ad938ab/volumes"
Nov 03 12:15:20.940810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.939984    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"
Nov 03 12:15:20.940810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.939998    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2" event={"ID":"8bb87988-b857-492f-afde-b13d2e744cbb","Type":"ContainerDied","Data":"4e33cb2e6119e31c2bc4adcd5b73679822fe1a40dea0cf722e5556e0154ed9b3"}
Nov 03 12:15:20.940810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.940154    4546 scope.go:122] "RemoveContainer" containerID="a44b9a41f2b0b4d1486fb353b438b03184a99d3c1995105474ffded553a8e407"
Nov 03 12:15:20.943361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.943333    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" event={"ID":"a995bae5-79c9-4200-921e-3940efb438d0","Type":"ContainerStarted","Data":"c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"}
Nov 03 12:15:20.943474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.943383    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" event={"ID":"a995bae5-79c9-4200-921e-3940efb438d0","Type":"ContainerStarted","Data":"351af2ec18fe5eb770bc00c885118b1784a856ff99266bb5e91914bb5c5bfcb0"}
Nov 03 12:15:20.950813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.950715    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr" event={"ID":"6e59ccbc-3318-44a5-9323-c9a077a213e2","Type":"ContainerDied","Data":"b8af0261574d4ac4722ef68c40048dd3e0311f7d35f901084db5a62dc24fa476"}
Nov 03 12:15:20.950925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.950831    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"
Nov 03 12:15:20.953545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.953521    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"]
Nov 03 12:15:20.955737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.955721    4546 scope.go:122] "RemoveContainer" containerID="7c479c59a4024c7415511fdd2b4c2f68f4f1b4a34919d24cc75fd59706fbc3fd"
Nov 03 12:15:20.958065 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:20.958044    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffffeda_ac8d_4c56_9cea_90eeac8842c8.slice/crio-f3e2dd263ee6bfe7b7fc64c6c8c395540d71c32d807db3800db9f7add3e19972 WatchSource:0}: Error finding container f3e2dd263ee6bfe7b7fc64c6c8c395540d71c32d807db3800db9f7add3e19972: Status 404 returned error can't find the container with id f3e2dd263ee6bfe7b7fc64c6c8c395540d71c32d807db3800db9f7add3e19972
Nov 03 12:15:20.960414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.960388    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:15:20.960493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.960456    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:15:20.960493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:20.960487    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:15:21.075343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.075310    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"]
Nov 03 12:15:21.096497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.096470    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"]
Nov 03 12:15:21.108576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.108543    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895135-scdcr"]
Nov 03 12:15:21.215274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.215247    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"]
Nov 03 12:15:21.222390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.222363    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895135-9q4q2"]
Nov 03 12:15:21.263077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.263055    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:21.330036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.329981    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podStartSLOduration=2.32996624 podStartE2EDuration="2.32996624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:21.323472202 +0000 UTC m=+1462.895056148" watchObservedRunningTime="2026-11-03 12:15:21.32996624 +0000 UTC m=+1462.901550151"
Nov 03 12:15:21.364725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.364687    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca\") pod \"5d0424de-8707-43e7-a16e-291a166615c7\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") "
Nov 03 12:15:21.364991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.364972    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5\" (UniqueName: \"kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5\") pod \"5d0424de-8707-43e7-a16e-291a166615c7\" (UID: \"5d0424de-8707-43e7-a16e-291a166615c7\") "
Nov 03 12:15:21.365085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.365065    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca" pod "5d0424de-8707-43e7-a16e-291a166615c7" (UID: "5d0424de-8707-43e7-a16e-291a166615c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:21.365702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.365681    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0424de-8707-43e7-a16e-291a166615c7-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:21.367162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.367144    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5" pod "5d0424de-8707-43e7-a16e-291a166615c7" (UID: "5d0424de-8707-43e7-a16e-291a166615c7"). InnerVolumeSpecName "kube-api-access-gv4b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:21.497022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.481419    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv4b5\" (UniqueName: \"kubernetes.io/projected/5d0424de-8707-43e7-a16e-291a166615c7-kube-api-access-gv4b5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:21.705018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.704986    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:21.735344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.735312    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:21.959384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.959285    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" event={"ID":"effffeda-ac8d-4c56-9cea-90eeac8842c8","Type":"ContainerStarted","Data":"c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf"}
Nov 03 12:15:21.959384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.959345    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" event={"ID":"effffeda-ac8d-4c56-9cea-90eeac8842c8","Type":"ContainerStarted","Data":"f3e2dd263ee6bfe7b7fc64c6c8c395540d71c32d807db3800db9f7add3e19972"}
Nov 03 12:15:21.960787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.960743    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r" event={"ID":"5d0424de-8707-43e7-a16e-291a166615c7","Type":"ContainerDied","Data":"ec86cde9de2dbdc7dabc93a5fe4f2957e1039f3352c427680ecebbd3397d3c71"}
Nov 03 12:15:21.960875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.960812    4546 scope.go:122] "RemoveContainer" containerID="51de73bec4f3b351f006da299b097a09be54bd7da9d4329b682da5968200f67e"
Nov 03 12:15:21.960875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.960769    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"
Nov 03 12:15:21.963491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.963469    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:21.964720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:21.964703    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:15:22.092448 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.092401    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podStartSLOduration=3.092387311 podStartE2EDuration="3.092387311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:22.088487816 +0000 UTC m=+1463.660071750" watchObservedRunningTime="2026-11-03 12:15:22.092387311 +0000 UTC m=+1463.663971245"
Nov 03 12:15:22.196741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.196704    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"]
Nov 03 12:15:22.201264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.201237    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895135-jhx6r"]
Nov 03 12:15:22.560141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.560113    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:22.561429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.561402    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 12:15:22.561559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.561463    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 12:15:22.576081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.576056    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0424de-8707-43e7-a16e-291a166615c7" path="/var/lib/kubelet/pods/5d0424de-8707-43e7-a16e-291a166615c7/volumes"
Nov 03 12:15:22.576470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.576457    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" path="/var/lib/kubelet/pods/6e59ccbc-3318-44a5-9323-c9a077a213e2/volumes"
Nov 03 12:15:22.576943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:22.576928    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" path="/var/lib/kubelet/pods/8bb87988-b857-492f-afde-b13d2e744cbb/volumes"
Nov 03 12:15:23.560279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:23.560246    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 12:15:23.560675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:23.560318    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 12:15:24.559627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:24.559590    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 12:15:24.559838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:24.559655    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 12:15:25.560288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:25.560251    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 12:15:25.560772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:25.560315    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 12:15:26.559821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:26.559786    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 12:15:26.559995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:26.559852    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 12:15:27.572801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.572743    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 12:15:27.573433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.573400    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.964379    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"]
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965374    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965396    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965419    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965426    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965442    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965450    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965463    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965470    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965503    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965510    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965521    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965528    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965568    4546 cpu_manager.go:414] "containerMap: removing container" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965575    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965717    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965729    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965760    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965774    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965785    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e01632-9a36-49e9-9aad-cd3f19e995ba" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965797    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965806    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.965838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.965815    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:27.983781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:27.982158    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.068525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.059195    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"]
Nov 03 12:15:28.068889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.065205    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") " pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.069307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.069284    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpj6\" (UniqueName: \"kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6\") pod \"payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") " pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.069553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.066091    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-913a17\""
Nov 03 12:15:28.069654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.065390    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-913a17-dockercfg-69tfk\""
Nov 03 12:15:28.122118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.121969    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"]
Nov 03 12:15:28.122989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.122962    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:28.122989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.122986    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:28.123200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.123012    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:28.123200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.123020    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb87988-b857-492f-afde-b13d2e744cbb" containerName="cleanup"
Nov 03 12:15:28.123200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.123052    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:28.123200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.123060    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:28.123200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.123197    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e59ccbc-3318-44a5-9323-c9a077a213e2" containerName="cleanup"
Nov 03 12:15:28.123449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.123208    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0424de-8707-43e7-a16e-291a166615c7" containerName="cleanup"
Nov 03 12:15:28.127058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.127033    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.172728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.172697    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") " pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.172970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.172855    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpj6\" (UniqueName: \"kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6\") pod \"payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") " pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.174634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.174049    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") " pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.180537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.180509    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"]
Nov 03 12:15:28.228575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.228481    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpj6\" (UniqueName: \"kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6\") pod \"payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") " pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.278781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.276641    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.278781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.276914    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4jt\" (UniqueName: \"kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt\") pod \"payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.340050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.340015    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:28.378171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.378128    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.378382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.378360    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4jt\" (UniqueName: \"kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt\") pod \"payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.378862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.378837    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.431147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.431111    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4jt\" (UniqueName: \"kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt\") pod \"payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.453614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.453574    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:28.564926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.564889    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:28.579642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.579609    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:28.580105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.579715    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 12:15:28.710898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.710864    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"]
Nov 03 12:15:28.747194 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:28.747161    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a8de4e_ba60_4c90_8def_64ce7046318e.slice/crio-84324242d1d6f0189f26ac4a5e2d68d92071885e5567eba17a21f7e540175bfd WatchSource:0}: Error finding container 84324242d1d6f0189f26ac4a5e2d68d92071885e5567eba17a21f7e540175bfd: Status 404 returned error can't find the container with id 84324242d1d6f0189f26ac4a5e2d68d92071885e5567eba17a21f7e540175bfd
Nov 03 12:15:28.831127 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:28.831084    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d3dd9c_5d87_42a0_90d8_2248e9e5dca8.slice/crio-a493589549d2e590b361b7d849f4e5bdb0cd4ae193c67a891cf81d9a151b5ea9 WatchSource:0}: Error finding container a493589549d2e590b361b7d849f4e5bdb0cd4ae193c67a891cf81d9a151b5ea9: Status 404 returned error can't find the container with id a493589549d2e590b361b7d849f4e5bdb0cd4ae193c67a891cf81d9a151b5ea9
Nov 03 12:15:28.832819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:28.832789    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"]
Nov 03 12:15:29.091375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.091277    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6" event={"ID":"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8","Type":"ContainerStarted","Data":"d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f"}
Nov 03 12:15:29.091375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.091340    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6" event={"ID":"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8","Type":"ContainerStarted","Data":"a493589549d2e590b361b7d849f4e5bdb0cd4ae193c67a891cf81d9a151b5ea9"}
Nov 03 12:15:29.091375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.091367    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:29.093400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.093373    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg" event={"ID":"02a8de4e-ba60-4c90-8def-64ce7046318e","Type":"ContainerStarted","Data":"71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887"}
Nov 03 12:15:29.093545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.093527    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:29.093589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.093570    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg" event={"ID":"02a8de4e-ba60-4c90-8def-64ce7046318e","Type":"ContainerStarted","Data":"84324242d1d6f0189f26ac4a5e2d68d92071885e5567eba17a21f7e540175bfd"}
Nov 03 12:15:29.201126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.201063    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6" podStartSLOduration=1.201045777 podStartE2EDuration="1.201045777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:29.196794688 +0000 UTC m=+1470.768378600" watchObservedRunningTime="2026-11-03 12:15:29.201045777 +0000 UTC m=+1470.772629690"
Nov 03 12:15:29.302018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:29.301955    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg" podStartSLOduration=2.3019405920000002 podStartE2EDuration="2.301940592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:29.301183511 +0000 UTC m=+1470.872767427" watchObservedRunningTime="2026-11-03 12:15:29.301940592 +0000 UTC m=+1470.873524504"
Nov 03 12:15:32.448727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.448693    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"]
Nov 03 12:15:32.453279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.453253    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.510879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.510848    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"]
Nov 03 12:15:32.551101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.549014    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-931523\""
Nov 03 12:15:32.551101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.549237    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-931523-dockercfg-lpgb9\""
Nov 03 12:15:32.647385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.640443    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v28c\" (UniqueName: \"kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c\") pod \"payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.657772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.647880    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.698117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.698083    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"]
Nov 03 12:15:32.702834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.702768    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"]
Nov 03 12:15:32.702957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.702833    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:32.759293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.759192    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v28c\" (UniqueName: \"kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c\") pod \"payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.759574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.759544    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.760353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.760327    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.813024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.812986    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v28c\" (UniqueName: \"kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c\") pod \"payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:32.861350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.861315    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8g7\" (UniqueName: \"kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7\") pod \"payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") " pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:32.861499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.861366    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") " pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:32.963818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.963792    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8g7\" (UniqueName: \"kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7\") pod \"payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") " pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:32.964041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.963839    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") " pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:32.964507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:32.964486    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") " pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:33.016784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:33.016741    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8g7\" (UniqueName: \"kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7\") pod \"payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") " pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:33.072101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:33.072069    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:33.317063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:33.316965    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:33.447684 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:33.447640    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d8fa43_9174_4a27_87a7_4f9db69aca42.slice/crio-fcb1f0d645f3e21b6d2635f8a62af6b837d72efa4490750a27cefe2eb3eba532 WatchSource:0}: Error finding container fcb1f0d645f3e21b6d2635f8a62af6b837d72efa4490750a27cefe2eb3eba532: Status 404 returned error can't find the container with id fcb1f0d645f3e21b6d2635f8a62af6b837d72efa4490750a27cefe2eb3eba532
Nov 03 12:15:33.450450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:33.450421    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"]
Nov 03 12:15:33.687273 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:33.687245    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca31a82_9146_4937_9e94_a56e03c81dc8.slice/crio-685801ede449f6a5b424ec352c369c93399d5960786f9df4c6f26fd9a7ad8595 WatchSource:0}: Error finding container 685801ede449f6a5b424ec352c369c93399d5960786f9df4c6f26fd9a7ad8595: Status 404 returned error can't find the container with id 685801ede449f6a5b424ec352c369c93399d5960786f9df4c6f26fd9a7ad8595
Nov 03 12:15:33.688089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:33.688019    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"]
Nov 03 12:15:34.138489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.138438    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp" event={"ID":"5ca31a82-9146-4937-9e94-a56e03c81dc8","Type":"ContainerStarted","Data":"cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce"}
Nov 03 12:15:34.138794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.138538    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp" event={"ID":"5ca31a82-9146-4937-9e94-a56e03c81dc8","Type":"ContainerStarted","Data":"685801ede449f6a5b424ec352c369c93399d5960786f9df4c6f26fd9a7ad8595"}
Nov 03 12:15:34.138900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.138873    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:34.140845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.140797    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc" event={"ID":"68d8fa43-9174-4a27-87a7-4f9db69aca42","Type":"ContainerStarted","Data":"a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100"}
Nov 03 12:15:34.141028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.140866    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc" event={"ID":"68d8fa43-9174-4a27-87a7-4f9db69aca42","Type":"ContainerStarted","Data":"fcb1f0d645f3e21b6d2635f8a62af6b837d72efa4490750a27cefe2eb3eba532"}
Nov 03 12:15:34.141028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.140935    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:34.350148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.348561    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp" podStartSLOduration=2.348541099 podStartE2EDuration="2.348541099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:34.249216131 +0000 UTC m=+1475.820800044" watchObservedRunningTime="2026-11-03 12:15:34.348541099 +0000 UTC m=+1475.920125028"
Nov 03 12:15:34.402136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:34.402018    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc" podStartSLOduration=2.402000023 podStartE2EDuration="2.402000023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:34.398462297 +0000 UTC m=+1475.970046203" watchObservedRunningTime="2026-11-03 12:15:34.402000023 +0000 UTC m=+1475.973583933"
Nov 03 12:15:37.652554 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:15:37.652479    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:15:40.104505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:40.104472    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:15:40.105089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:40.104612    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:15:45.153579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:45.153544    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:15:45.154049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:45.153632    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:15:46.481441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.481412    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"]
Nov 03 12:15:46.486395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.486375    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.605774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.592799    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f6c034\""
Nov 03 12:15:46.605774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.593036    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f6c034-dockercfg-7f9cg\""
Nov 03 12:15:46.685688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.671104    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"]
Nov 03 12:15:46.702965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.702926    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"]
Nov 03 12:15:46.703475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.702974    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqvr\" (UniqueName: \"kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr\") pod \"payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.703475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.703040    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.708727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.708704    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:46.710654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.710564    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"]
Nov 03 12:15:46.804978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.804888    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqvr\" (UniqueName: \"kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr\") pod \"payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.804978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.804965    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:46.805212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.805016    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.805273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.805232    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvmr\" (UniqueName: \"kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr\") pod \"payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:46.805712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.805689    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.863911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.863879    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqvr\" (UniqueName: \"kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr\") pod \"payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:46.906864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.906820    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvmr\" (UniqueName: \"kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr\") pod \"payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:46.907035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.907004    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:46.907775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.907737    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:46.967176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:46.967141    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvmr\" (UniqueName: \"kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr\") pod \"payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:47.022876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:47.022843    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:47.105082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:47.105050    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:47.393261 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:47.393228    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a533ac7_f535_4725_bb9d_c7f3a76f44fd.slice/crio-d43f064692a9a44af80420ff37cac8fa1b651f9aad814293fd6aa18dec7623e0 WatchSource:0}: Error finding container d43f064692a9a44af80420ff37cac8fa1b651f9aad814293fd6aa18dec7623e0: Status 404 returned error can't find the container with id d43f064692a9a44af80420ff37cac8fa1b651f9aad814293fd6aa18dec7623e0
Nov 03 12:15:47.393949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:47.393911    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"]
Nov 03 12:15:47.491280 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:15:47.491253    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f10df6_086e_475b_bbd5_624329ba890e.slice/crio-71ec4943fced834f7360d19fcc4862b859e9bba187aa2e9f6934e252b998a034 WatchSource:0}: Error finding container 71ec4943fced834f7360d19fcc4862b859e9bba187aa2e9f6934e252b998a034: Status 404 returned error can't find the container with id 71ec4943fced834f7360d19fcc4862b859e9bba187aa2e9f6934e252b998a034
Nov 03 12:15:47.492028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:47.491917    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"]
Nov 03 12:15:48.046337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.046306    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"]
Nov 03 12:15:48.051002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.050969    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738265a-8kztq"]
Nov 03 12:15:48.267301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.267270    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh" event={"ID":"2a533ac7-f535-4725-bb9d-c7f3a76f44fd","Type":"ContainerStarted","Data":"b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a"}
Nov 03 12:15:48.267502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.267322    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh" event={"ID":"2a533ac7-f535-4725-bb9d-c7f3a76f44fd","Type":"ContainerStarted","Data":"d43f064692a9a44af80420ff37cac8fa1b651f9aad814293fd6aa18dec7623e0"}
Nov 03 12:15:48.267502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.267387    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:15:48.269288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.269258    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9" event={"ID":"93f10df6-086e-475b-bbd5-624329ba890e","Type":"ContainerStarted","Data":"ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6"}
Nov 03 12:15:48.269423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.269307    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9" event={"ID":"93f10df6-086e-475b-bbd5-624329ba890e","Type":"ContainerStarted","Data":"71ec4943fced834f7360d19fcc4862b859e9bba187aa2e9f6934e252b998a034"}
Nov 03 12:15:48.269568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.269535    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:48.373909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.373807    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh" podStartSLOduration=2.37378888 podStartE2EDuration="2.37378888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:48.3721639 +0000 UTC m=+1489.943747810" watchObservedRunningTime="2026-11-03 12:15:48.37378888 +0000 UTC m=+1489.945372795"
Nov 03 12:15:48.477011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.476962    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9" podStartSLOduration=2.476949197 podStartE2EDuration="2.476949197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:48.475889102 +0000 UTC m=+1490.047473013" watchObservedRunningTime="2026-11-03 12:15:48.476949197 +0000 UTC m=+1490.048533109"
Nov 03 12:15:48.578623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:48.578588    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6588442e-0477-4d80-93b9-5a103f47bd0d" path="/var/lib/kubelet/pods/6588442e-0477-4d80-93b9-5a103f47bd0d/volumes"
Nov 03 12:15:59.279441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:59.279416    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:15:59.279887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:15:59.279797    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:16:04.050213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:04.050171    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"]
Nov 03 12:16:04.093083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:04.093047    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc44345f-bwrs4"]
Nov 03 12:16:04.575139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:04.575104    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba14fb1-fa7c-4d32-a530-0baee6adb556" path="/var/lib/kubelet/pods/7ba14fb1-fa7c-4d32-a530-0baee6adb556/volumes"
Nov 03 12:16:07.652690 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:16:07.652652    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:16:20.046648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:20.046612    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"]
Nov 03 12:16:20.093034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:20.093004    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"]
Nov 03 12:16:20.097264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:20.097239    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88ac920c-kcvf4"]
Nov 03 12:16:20.101139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:20.101119    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f292f59-8z9fm"]
Nov 03 12:16:20.575587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:20.575554    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56446e5e-cfe6-4e01-b743-917bd6e6b27e" path="/var/lib/kubelet/pods/56446e5e-cfe6-4e01-b743-917bd6e6b27e/volumes"
Nov 03 12:16:20.576149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:20.576132    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd904dc7-4d52-4833-a4ea-268e492b7a6c" path="/var/lib/kubelet/pods/dd904dc7-4d52-4833-a4ea-268e492b7a6c/volumes"
Nov 03 12:16:21.417819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.417784    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"]
Nov 03 12:16:21.423548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.423528    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.475394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.475363    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"]
Nov 03 12:16:21.495297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.495262    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.495534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.495513    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.495778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.495732    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swr26\" (UniqueName: \"kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.516441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.516407    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:16:21.516441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.516438    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:16:21.519784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.519727    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:16:21.519915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.519853    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:16:21.598044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.598008    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swr26\" (UniqueName: \"kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.598241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.598059    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.598241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.598224    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.598764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.598726    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.600688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.600658    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.650915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.650889    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swr26\" (UniqueName: \"kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:21.737355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:21.737319    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:22.106619 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:16:22.106587    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13bd48c_139e_410d_9ffc_95f8c7bc5879.slice/crio-17260f4f3d92a40541ec938c05959240976b5db3156bdf76b95a50684d571c10 WatchSource:0}: Error finding container 17260f4f3d92a40541ec938c05959240976b5db3156bdf76b95a50684d571c10: Status 404 returned error can't find the container with id 17260f4f3d92a40541ec938c05959240976b5db3156bdf76b95a50684d571c10
Nov 03 12:16:22.106920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:22.106900    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"]
Nov 03 12:16:22.533736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:22.533700    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk" event={"ID":"e13bd48c-139e-410d-9ffc-95f8c7bc5879","Type":"ContainerStarted","Data":"0d6927ffcdf4b79a3a55b243a4ce930f44bdace8020e12302abf677def4ffa94"}
Nov 03 12:16:22.534173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:22.533778    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk" event={"ID":"e13bd48c-139e-410d-9ffc-95f8c7bc5879","Type":"ContainerStarted","Data":"17260f4f3d92a40541ec938c05959240976b5db3156bdf76b95a50684d571c10"}
Nov 03 12:16:22.639949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:22.639897    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk" podStartSLOduration=1.639883421 podStartE2EDuration="1.639883421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:22.637828678 +0000 UTC m=+1524.209412607" watchObservedRunningTime="2026-11-03 12:16:22.639883421 +0000 UTC m=+1524.211467348"
Nov 03 12:16:23.543119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.543089    4546 generic.go:356] "Generic (PLEG): container finished" podID="e13bd48c-139e-410d-9ffc-95f8c7bc5879" containerID="0d6927ffcdf4b79a3a55b243a4ce930f44bdace8020e12302abf677def4ffa94" exitCode=0
Nov 03 12:16:23.543523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.543177    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk" event={"ID":"e13bd48c-139e-410d-9ffc-95f8c7bc5879","Type":"ContainerDied","Data":"0d6927ffcdf4b79a3a55b243a4ce930f44bdace8020e12302abf677def4ffa94"}
Nov 03 12:16:23.643839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.643807    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"]
Nov 03 12:16:23.648169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.648149    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.702435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.702403    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"]
Nov 03 12:16:23.831389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.831304    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcpw\" (UniqueName: \"kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.831389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.831376    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.831590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.831452    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.932815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.932776    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcpw\" (UniqueName: \"kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.932998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.932863    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.932998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.932887    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.933692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.933670    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.935471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.935451    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:23.985386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:23.985358    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcpw\" (UniqueName: \"kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw\") pod \"maas-api-revoke-sub-3a037e0b-snr9r\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:24.262822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.262743    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:24.643499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.643466    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"]
Nov 03 12:16:24.701591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.701566    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:24.847401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.847363    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26\" (UniqueName: \"kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26\") pod \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") "
Nov 03 12:16:24.847613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.847416    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca\") pod \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") "
Nov 03 12:16:24.847818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.847795    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca\") pod \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\" (UID: \"e13bd48c-139e-410d-9ffc-95f8c7bc5879\") "
Nov 03 12:16:24.848601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.848580    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca" pod "e13bd48c-139e-410d-9ffc-95f8c7bc5879" (UID: "e13bd48c-139e-410d-9ffc-95f8c7bc5879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:16:24.850375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.850342    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26" pod "e13bd48c-139e-410d-9ffc-95f8c7bc5879" (UID: "e13bd48c-139e-410d-9ffc-95f8c7bc5879"). InnerVolumeSpecName "kube-api-access-swr26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:16:24.850511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.850488    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca" pod "e13bd48c-139e-410d-9ffc-95f8c7bc5879" (UID: "e13bd48c-139e-410d-9ffc-95f8c7bc5879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:16:24.949458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.949429    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:24.949588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.949460    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swr26\" (UniqueName: \"kubernetes.io/projected/e13bd48c-139e-410d-9ffc-95f8c7bc5879-kube-api-access-swr26\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:24.949588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:24.949475    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13bd48c-139e-410d-9ffc-95f8c7bc5879-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:25.559997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:25.559970    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"
Nov 03 12:16:25.560207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:25.560008    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk" event={"ID":"e13bd48c-139e-410d-9ffc-95f8c7bc5879","Type":"ContainerDied","Data":"17260f4f3d92a40541ec938c05959240976b5db3156bdf76b95a50684d571c10"}
Nov 03 12:16:25.560207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:25.560060    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17260f4f3d92a40541ec938c05959240976b5db3156bdf76b95a50684d571c10"
Nov 03 12:16:25.562106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:25.562074    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r" event={"ID":"b316da01-5d90-4e6d-b8eb-edfffae718c2","Type":"ContainerStarted","Data":"123d713a70250fc1d99ebbcb71becbcfa92fd192147d28fc444a57bb12a7336d"}
Nov 03 12:16:25.562240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:25.562122    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r" event={"ID":"b316da01-5d90-4e6d-b8eb-edfffae718c2","Type":"ContainerStarted","Data":"301b464c1e583d87240f776ff6514376a53b5355bae522738493b17b60b642b3"}
Nov 03 12:16:25.668415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:25.668367    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r" podStartSLOduration=2.668355077 podStartE2EDuration="2.668355077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:25.667556853 +0000 UTC m=+1527.239140776" watchObservedRunningTime="2026-11-03 12:16:25.668355077 +0000 UTC m=+1527.239938989"
Nov 03 12:16:26.572243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:26.572210    4546 generic.go:356] "Generic (PLEG): container finished" podID="b316da01-5d90-4e6d-b8eb-edfffae718c2" containerID="123d713a70250fc1d99ebbcb71becbcfa92fd192147d28fc444a57bb12a7336d" exitCode=0
Nov 03 12:16:26.575588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:26.575557    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r" event={"ID":"b316da01-5d90-4e6d-b8eb-edfffae718c2","Type":"ContainerDied","Data":"123d713a70250fc1d99ebbcb71becbcfa92fd192147d28fc444a57bb12a7336d"}
Nov 03 12:16:27.748670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.748643    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:27.782177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.782149    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca\") pod \"b316da01-5d90-4e6d-b8eb-edfffae718c2\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") "
Nov 03 12:16:27.782444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.782424    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw\" (UniqueName: \"kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw\") pod \"b316da01-5d90-4e6d-b8eb-edfffae718c2\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") "
Nov 03 12:16:27.782502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.782480    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca" pod "b316da01-5d90-4e6d-b8eb-edfffae718c2" (UID: "b316da01-5d90-4e6d-b8eb-edfffae718c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:16:27.782555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.782502    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca\") pod \"b316da01-5d90-4e6d-b8eb-edfffae718c2\" (UID: \"b316da01-5d90-4e6d-b8eb-edfffae718c2\") "
Nov 03 12:16:27.783160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.783095    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:27.784642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.784620    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw" pod "b316da01-5d90-4e6d-b8eb-edfffae718c2" (UID: "b316da01-5d90-4e6d-b8eb-edfffae718c2"). InnerVolumeSpecName "kube-api-access-cfcpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:16:27.784744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.784667    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca" pod "b316da01-5d90-4e6d-b8eb-edfffae718c2" (UID: "b316da01-5d90-4e6d-b8eb-edfffae718c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:16:27.885403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.885319    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcpw\" (UniqueName: \"kubernetes.io/projected/b316da01-5d90-4e6d-b8eb-edfffae718c2-kube-api-access-cfcpw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:27.885403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:27.885362    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b316da01-5d90-4e6d-b8eb-edfffae718c2-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:28.588131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:28.588097    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"
Nov 03 12:16:28.588306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:28.588132    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r" event={"ID":"b316da01-5d90-4e6d-b8eb-edfffae718c2","Type":"ContainerDied","Data":"301b464c1e583d87240f776ff6514376a53b5355bae522738493b17b60b642b3"}
Nov 03 12:16:28.588306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:28.588188    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="301b464c1e583d87240f776ff6514376a53b5355bae522738493b17b60b642b3"
Nov 03 12:16:34.062008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:34.061962    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"]
Nov 03 12:16:34.095014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:34.094978    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0aaf99e-tcpmj"]
Nov 03 12:16:34.581935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:34.581899    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c35050-7a2a-4c2d-9bee-9ad19d1fbef4" path="/var/lib/kubelet/pods/99c35050-7a2a-4c2d-9bee-9ad19d1fbef4/volumes"
Nov 03 12:16:37.045401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:37.045370    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"]
Nov 03 12:16:37.090872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:37.090837    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d684072e-nstls"]
Nov 03 12:16:37.653627 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:16:37.653541    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:16:38.578869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:38.578834    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91f7944-8518-4982-9545-a76252c09237" path="/var/lib/kubelet/pods/c91f7944-8518-4982-9545-a76252c09237/volumes"
Nov 03 12:16:41.726015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:41.725985    4546 generic.go:356] "Generic (PLEG): container finished" podID="a995bae5-79c9-4200-921e-3940efb438d0" containerID="c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded" exitCode=0
Nov 03 12:16:41.726478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:41.726062    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" event={"ID":"a995bae5-79c9-4200-921e-3940efb438d0","Type":"ContainerDied","Data":"c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"}
Nov 03 12:16:41.726527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:41.726505    4546 scope.go:122] "RemoveContainer" containerID="c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"
Nov 03 12:16:41.728180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:41.728150    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:16:41.728289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:41.728237    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:16:41.728356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:41.728287    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:16:42.740344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:42.740308    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" event={"ID":"a995bae5-79c9-4200-921e-3940efb438d0","Type":"ContainerStarted","Data":"0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb"}
Nov 03 12:16:43.705466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:43.705431    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:16:43.705646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:43.705628    4546 patch_prober.go:29] interesting pod/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:16:43.705705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:43.705680    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 12:16:44.704874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:44.704840    4546 patch_prober.go:29] interesting pod/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:16:44.705273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:44.704899    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 12:16:45.705329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:45.705293    4546 patch_prober.go:29] interesting pod/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:16:45.705720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:45.705375    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 12:16:46.705140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:46.705108    4546 patch_prober.go:29] interesting pod/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:16:46.705339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:46.705165    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 12:16:47.046284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:47.046249    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"]
Nov 03 12:16:47.092598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:47.092569    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da2654-pwmc2"]
Nov 03 12:16:47.705293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:47.705257    4546 patch_prober.go:29] interesting pod/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 12:16:47.705605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:47.705561    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 12:16:48.045117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.045082    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"]
Nov 03 12:16:48.089181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.089151    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d662fe04-nm7fj"]
Nov 03 12:16:48.574910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.574877    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6" path="/var/lib/kubelet/pods/2b1cbe70-1a7d-4b6f-9888-f3f1e36e7ba6/volumes"
Nov 03 12:16:48.575393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.575378    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7196841-9f2e-4cee-a435-509f70d34bea" path="/var/lib/kubelet/pods/e7196841-9f2e-4cee-a435-509f70d34bea/volumes"
Nov 03 12:16:48.710201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.710170    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:16:48.710676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.710660    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:16:48.711432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:48.711417    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:16:50.585469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.585427    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"]
Nov 03 12:16:50.586445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.586421    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b316da01-5d90-4e6d-b8eb-edfffae718c2" containerName="revoke-keys"
Nov 03 12:16:50.586445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.586444    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b316da01-5d90-4e6d-b8eb-edfffae718c2" containerName="revoke-keys"
Nov 03 12:16:50.586578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.586479    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e13bd48c-139e-410d-9ffc-95f8c7bc5879" containerName="revoke-keys"
Nov 03 12:16:50.586578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.586485    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13bd48c-139e-410d-9ffc-95f8c7bc5879" containerName="revoke-keys"
Nov 03 12:16:50.586684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.586642    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b316da01-5d90-4e6d-b8eb-edfffae718c2" containerName="revoke-keys"
Nov 03 12:16:50.586684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.586657    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13bd48c-139e-410d-9ffc-95f8c7bc5879" containerName="revoke-keys"
Nov 03 12:16:50.590761 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.590731    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.686794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.686719    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"]
Nov 03 12:16:50.695019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.694972    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695043    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695171    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695232    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695426    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695573    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695704    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695738    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.695819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.695778    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8z6\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797181    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797254    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797369    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797415    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8z6\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797556    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797604    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797630    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797668    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797679    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797715    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797862    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.797977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.797924    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.798071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.798033    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.798486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.798452    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.799887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.799859    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.800701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.800677    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.850697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.850617    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.850907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.850881    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8z6\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:50.905498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:50.905460    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:51.332478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.332438    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"]
Nov 03 12:16:51.340520 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:16:51.340480    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec9fec2_2b54_42f0_bbb8_f85f8233b84f.slice/crio-8d1a66f5b1825760c3e4bbdcfeee3c1a55cb158d9fbffa38b35ce73a2d94504e WatchSource:0}: Error finding container 8d1a66f5b1825760c3e4bbdcfeee3c1a55cb158d9fbffa38b35ce73a2d94504e: Status 404 returned error can't find the container with id 8d1a66f5b1825760c3e4bbdcfeee3c1a55cb158d9fbffa38b35ce73a2d94504e
Nov 03 12:16:51.343007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.342978    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:16:51.343060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.343042    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:16:51.343095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.343070    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:16:51.815927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.815892    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" event={"ID":"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f","Type":"ContainerStarted","Data":"466fc8f0c46162286c58091b7531689b0289e638cb217761b186bf3884d9f683"}
Nov 03 12:16:51.816387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.815946    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" event={"ID":"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f","Type":"ContainerStarted","Data":"8d1a66f5b1825760c3e4bbdcfeee3c1a55cb158d9fbffa38b35ce73a2d94504e"}
Nov 03 12:16:51.906640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.906596    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:51.973938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:51.973877    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podStartSLOduration=1.9738633559999998 podStartE2EDuration="1.973863356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:51.923385113 +0000 UTC m=+1553.494969047" watchObservedRunningTime="2026-11-03 12:16:51.973863356 +0000 UTC m=+1553.545447265"
Nov 03 12:16:52.929774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:52.928897    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:52.929774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:52.928969    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:53.906257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:53.906220    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:53.906450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:53.906275    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:54.906551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:54.906519    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:54.907041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:54.906584    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:55.906027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:55.905990    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:55.906225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:55.906055    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:56.906300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:56.906258    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:56.906705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:56.906318    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:57.906446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:57.906399    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:57.906992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:57.906472    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:58.906400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:58.906359    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 12:16:58.906555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:58.906421    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 12:16:59.138399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:59.138362    4546 scope.go:122] "RemoveContainer" containerID="e23489659d819e85a0b413c881973b6f826b4b11d3d24d226f3d5b0f81baa14c"
Nov 03 12:16:59.944437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:59.944401    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:59.966880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:59.966854    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:16:59.998107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:16:59.998076    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:17:04.722423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.722384    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"]
Nov 03 12:17:04.728901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.728880    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.766691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.766660    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqn2f\" (UniqueName: \"kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.766988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.766966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.767065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.767020    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.781418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.781393    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"]
Nov 03 12:17:04.821932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.821897    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:17:04.822085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.821997    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:17:04.823332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.823306    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:17:04.823512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.823496    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:17:04.867990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.867957    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.868136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.868001    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.868136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.868060    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn2f\" (UniqueName: \"kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.868641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.868617    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.870473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.870449    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:04.948784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:04.947465    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqn2f\" (UniqueName: \"kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f\") pod \"maas-api-revoke-sub-17fc161d-q9rtt\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:05.011825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.009431    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"]
Nov 03 12:17:05.017773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.017307    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.036271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.036224    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"]
Nov 03 12:17:05.053707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.043232    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:05.077908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.077797    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.077908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.077877    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.078178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.078026    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjfg\" (UniqueName: \"kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.093860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.082917    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"]
Nov 03 12:17:05.093860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.091287    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"]
Nov 03 12:17:05.099742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.099591    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcfe57f-gjg82"]
Nov 03 12:17:05.107020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.106968    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a3ae1e-jlwk7"]
Nov 03 12:17:05.186020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.184431    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.186020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.184516    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.186020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.184660    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjfg\" (UniqueName: \"kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.190810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.188578    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.190810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.189285    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.239873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.239838    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjfg\" (UniqueName: \"kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg\") pod \"maas-api-revoke-sub-78a7ffe0-mhvfc\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.355722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.355645    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:05.466459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.466425    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"]
Nov 03 12:17:05.466776 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:17:05.466736    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4a523a_0a1e_42d8_9c19_5561579e7d95.slice/crio-d8f7a87d151356ab1cee8960fead6f9670b01519f027ba26d22b177860e199ab WatchSource:0}: Error finding container d8f7a87d151356ab1cee8960fead6f9670b01519f027ba26d22b177860e199ab: Status 404 returned error can't find the container with id d8f7a87d151356ab1cee8960fead6f9670b01519f027ba26d22b177860e199ab
Nov 03 12:17:05.726514 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:17:05.726469    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84514f25_e4c6_477a_93b9_662eaa4ff408.slice/crio-7969c8eae0c08b0fc77680217440bedddf09c505e3f37c8376195f5d44b14010 WatchSource:0}: Error finding container 7969c8eae0c08b0fc77680217440bedddf09c505e3f37c8376195f5d44b14010: Status 404 returned error can't find the container with id 7969c8eae0c08b0fc77680217440bedddf09c505e3f37c8376195f5d44b14010
Nov 03 12:17:05.727972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:05.727715    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"]
Nov 03 12:17:06.018835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:06.018801    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt" event={"ID":"4b4a523a-0a1e-42d8-9c19-5561579e7d95","Type":"ContainerStarted","Data":"247013ab9eb3b42207a53409817ee489154487f7f1d7662044175f2c257b1601"}
Nov 03 12:17:06.019017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:06.018864    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt" event={"ID":"4b4a523a-0a1e-42d8-9c19-5561579e7d95","Type":"ContainerStarted","Data":"d8f7a87d151356ab1cee8960fead6f9670b01519f027ba26d22b177860e199ab"}
Nov 03 12:17:06.020346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:06.020306    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc" event={"ID":"84514f25-e4c6-477a-93b9-662eaa4ff408","Type":"ContainerStarted","Data":"7969c8eae0c08b0fc77680217440bedddf09c505e3f37c8376195f5d44b14010"}
Nov 03 12:17:06.125307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:06.125261    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt" podStartSLOduration=2.125247439 podStartE2EDuration="2.125247439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:06.123506185 +0000 UTC m=+1567.695090110" watchObservedRunningTime="2026-11-03 12:17:06.125247439 +0000 UTC m=+1567.696831361"
Nov 03 12:17:06.580328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:06.580287    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b746f53c-4a17-4ba5-8fdb-7868a9ad9193" path="/var/lib/kubelet/pods/b746f53c-4a17-4ba5-8fdb-7868a9ad9193/volumes"
Nov 03 12:17:06.580958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:06.580940    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd83587-b53a-4163-8230-2e560470c41d" path="/var/lib/kubelet/pods/cdd83587-b53a-4163-8230-2e560470c41d/volumes"
Nov 03 12:17:07.033259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:07.033220    4546 generic.go:356] "Generic (PLEG): container finished" podID="84514f25-e4c6-477a-93b9-662eaa4ff408" containerID="a2b83dea795c20134776f0d48abbd3736be0a9ad84cad47a0b01d9a0547fac73" exitCode=0
Nov 03 12:17:07.033803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:07.033303    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc" event={"ID":"84514f25-e4c6-477a-93b9-662eaa4ff408","Type":"ContainerDied","Data":"a2b83dea795c20134776f0d48abbd3736be0a9ad84cad47a0b01d9a0547fac73"}
Nov 03 12:17:07.035546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:07.035519    4546 generic.go:356] "Generic (PLEG): container finished" podID="4b4a523a-0a1e-42d8-9c19-5561579e7d95" containerID="247013ab9eb3b42207a53409817ee489154487f7f1d7662044175f2c257b1601" exitCode=0
Nov 03 12:17:07.035653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:07.035569    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt" event={"ID":"4b4a523a-0a1e-42d8-9c19-5561579e7d95","Type":"ContainerDied","Data":"247013ab9eb3b42207a53409817ee489154487f7f1d7662044175f2c257b1601"}
Nov 03 12:17:07.653931 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:07.653851    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:17:08.223559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.223530    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:08.261560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.261532    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:08.327070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.326990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca\") pod \"84514f25-e4c6-477a-93b9-662eaa4ff408\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") "
Nov 03 12:17:08.327070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327031    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f\" (UniqueName: \"kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f\") pod \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") "
Nov 03 12:17:08.327357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327075    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca\") pod \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") "
Nov 03 12:17:08.327357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327246    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca\") pod \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\" (UID: \"4b4a523a-0a1e-42d8-9c19-5561579e7d95\") "
Nov 03 12:17:08.327357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327350    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca\") pod \"84514f25-e4c6-477a-93b9-662eaa4ff408\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") "
Nov 03 12:17:08.327847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327487    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg\" (UniqueName: \"kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg\") pod \"84514f25-e4c6-477a-93b9-662eaa4ff408\" (UID: \"84514f25-e4c6-477a-93b9-662eaa4ff408\") "
Nov 03 12:17:08.327847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327483    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca" pod "4b4a523a-0a1e-42d8-9c19-5561579e7d95" (UID: "4b4a523a-0a1e-42d8-9c19-5561579e7d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:08.328095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.327915    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca" pod "84514f25-e4c6-477a-93b9-662eaa4ff408" (UID: "84514f25-e4c6-477a-93b9-662eaa4ff408"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:08.328488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.328433    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:08.328488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.328461    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:08.330490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.329882    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f" pod "4b4a523a-0a1e-42d8-9c19-5561579e7d95" (UID: "4b4a523a-0a1e-42d8-9c19-5561579e7d95"). InnerVolumeSpecName "kube-api-access-lqn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:08.330490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.330078    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca" pod "4b4a523a-0a1e-42d8-9c19-5561579e7d95" (UID: "4b4a523a-0a1e-42d8-9c19-5561579e7d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:08.330719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.330696    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg" pod "84514f25-e4c6-477a-93b9-662eaa4ff408" (UID: "84514f25-e4c6-477a-93b9-662eaa4ff408"). InnerVolumeSpecName "kube-api-access-6bjfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:08.331683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.331618    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca" pod "84514f25-e4c6-477a-93b9-662eaa4ff408" (UID: "84514f25-e4c6-477a-93b9-662eaa4ff408"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:08.430276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.430242    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bjfg\" (UniqueName: \"kubernetes.io/projected/84514f25-e4c6-477a-93b9-662eaa4ff408-kube-api-access-6bjfg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:08.430276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.430273    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84514f25-e4c6-477a-93b9-662eaa4ff408-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:08.430276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.430285    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqn2f\" (UniqueName: \"kubernetes.io/projected/4b4a523a-0a1e-42d8-9c19-5561579e7d95-kube-api-access-lqn2f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:08.430511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:08.430295    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4a523a-0a1e-42d8-9c19-5561579e7d95-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:09.052814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:09.052780    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"
Nov 03 12:17:09.052987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:09.052782    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt" event={"ID":"4b4a523a-0a1e-42d8-9c19-5561579e7d95","Type":"ContainerDied","Data":"d8f7a87d151356ab1cee8960fead6f9670b01519f027ba26d22b177860e199ab"}
Nov 03 12:17:09.052987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:09.052905    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f7a87d151356ab1cee8960fead6f9670b01519f027ba26d22b177860e199ab"
Nov 03 12:17:09.054295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:09.054269    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc" event={"ID":"84514f25-e4c6-477a-93b9-662eaa4ff408","Type":"ContainerDied","Data":"7969c8eae0c08b0fc77680217440bedddf09c505e3f37c8376195f5d44b14010"}
Nov 03 12:17:09.054429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:09.054297    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"
Nov 03 12:17:09.054429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:09.054309    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7969c8eae0c08b0fc77680217440bedddf09c505e3f37c8376195f5d44b14010"
Nov 03 12:17:14.225482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:14.225447    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"]
Nov 03 12:17:15.048296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:15.048258    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"]
Nov 03 12:17:15.052472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:15.052445    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc74b24-r7z24"]
Nov 03 12:17:15.056155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:15.056128    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"]
Nov 03 12:17:15.059591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:15.059573    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43afc493-jd9ss"]
Nov 03 12:17:16.574507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:16.574474    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e0fcf2-be21-4033-b808-90af548a0034" path="/var/lib/kubelet/pods/56e0fcf2-be21-4033-b808-90af548a0034/volumes"
Nov 03 12:17:16.575059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:16.575040    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2942f2-c3a6-4ae0-80c4-9f98f568e77a" path="/var/lib/kubelet/pods/af2942f2-c3a6-4ae0-80c4-9f98f568e77a/volumes"
Nov 03 12:17:19.226343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.226279    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj" podUID="1ba5dbef-8bb9-4fad-83d1-ce77a6570985" containerName="payload-pre-processing" containerID="cri-o://f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e" gracePeriod=25
Nov 03 12:17:19.503285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.503260    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:17:19.590651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.590620    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume\") pod \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") "
Nov 03 12:17:19.590849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.590665    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl\" (UniqueName: \"kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl\") pod \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\" (UID: \"1ba5dbef-8bb9-4fad-83d1-ce77a6570985\") "
Nov 03 12:17:19.591059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.591034    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume" pod "1ba5dbef-8bb9-4fad-83d1-ce77a6570985" (UID: "1ba5dbef-8bb9-4fad-83d1-ce77a6570985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:19.591381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.591362    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:19.592975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.592949    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl" pod "1ba5dbef-8bb9-4fad-83d1-ce77a6570985" (UID: "1ba5dbef-8bb9-4fad-83d1-ce77a6570985"). InnerVolumeSpecName "kube-api-access-546wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:19.692485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:19.692460    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-546wl\" (UniqueName: \"kubernetes.io/projected/1ba5dbef-8bb9-4fad-83d1-ce77a6570985-kube-api-access-546wl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:20.168146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.168110    4546 generic.go:356] "Generic (PLEG): container finished" podID="1ba5dbef-8bb9-4fad-83d1-ce77a6570985" containerID="f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e" exitCode=0
Nov 03 12:17:20.168146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.168162    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"
Nov 03 12:17:20.168463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.168301    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj" event={"ID":"1ba5dbef-8bb9-4fad-83d1-ce77a6570985","Type":"ContainerDied","Data":"f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e"}
Nov 03 12:17:20.168463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.168343    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj" event={"ID":"1ba5dbef-8bb9-4fad-83d1-ce77a6570985","Type":"ContainerDied","Data":"fd4628b48de4d0175ed36bf84a9441f822c7809db8a5c5cbc22d0df01d87c96f"}
Nov 03 12:17:20.168463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.168368    4546 scope.go:122] "RemoveContainer" containerID="f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e"
Nov 03 12:17:20.185732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.185707    4546 scope.go:122] "RemoveContainer" containerID="f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e"
Nov 03 12:17:20.186089 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:20.186061    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e\": container with ID starting with f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e not found: ID does not exist" containerID="f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e"
Nov 03 12:17:20.186212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.186093    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e"} err="failed to get container status \"f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e\": rpc error: code = NotFound desc = could not find container \"f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e\": container with ID starting with f74c79bb9daedce0088948612cd2d7d7032fa1751291daff7ebd33cb976d0a4e not found: ID does not exist"
Nov 03 12:17:20.324375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.324299    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"]
Nov 03 12:17:20.370126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.370097    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86d7ad-645b797767-rhwmj"]
Nov 03 12:17:20.578445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:20.578410    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba5dbef-8bb9-4fad-83d1-ce77a6570985" path="/var/lib/kubelet/pods/1ba5dbef-8bb9-4fad-83d1-ce77a6570985/volumes"
Nov 03 12:17:22.292046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:22.292007    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"]
Nov 03 12:17:22.395167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:22.395127    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"]
Nov 03 12:17:24.815788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.815723    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"]
Nov 03 12:17:24.816438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816422    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1ba5dbef-8bb9-4fad-83d1-ce77a6570985" containerName="payload-pre-processing"
Nov 03 12:17:24.816438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816437    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba5dbef-8bb9-4fad-83d1-ce77a6570985" containerName="payload-pre-processing"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816458    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4b4a523a-0a1e-42d8-9c19-5561579e7d95" containerName="revoke-keys"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816463    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4a523a-0a1e-42d8-9c19-5561579e7d95" containerName="revoke-keys"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816474    4546 cpu_manager.go:414] "containerMap: removing container" podUID="84514f25-e4c6-477a-93b9-662eaa4ff408" containerName="revoke-keys"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816479    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84514f25-e4c6-477a-93b9-662eaa4ff408" containerName="revoke-keys"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816578    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba5dbef-8bb9-4fad-83d1-ce77a6570985" containerName="payload-pre-processing"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816592    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4a523a-0a1e-42d8-9c19-5561579e7d95" containerName="revoke-keys"
Nov 03 12:17:24.816671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.816603    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="84514f25-e4c6-477a-93b9-662eaa4ff408" containerName="revoke-keys"
Nov 03 12:17:24.821933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.821915    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.870447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.870404    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8e8d12-openshift-default-dockercfg-kcj98\""
Nov 03 12:17:24.924293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.924259    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"]
Nov 03 12:17:24.939551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.939522    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.939729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.939564    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.939729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.939653    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.939873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.939830    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.939918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.939893    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.940052 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.940035    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gv7\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.940190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.940174    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.940239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.940216    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:24.940317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:24.940242    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.041954    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042016    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042108    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gv7\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042229    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042246    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042277    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042343    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042364    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042387    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.042520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042429    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.043051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.043006    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.043051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.043046    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.043253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.042968    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.043253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.043194    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.045799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.045775    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.046199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.046177    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.096674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.096591    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.096896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.096685    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gv7\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7\") pod \"e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") " pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.136603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.136522    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:25.569248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.569216    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"]
Nov 03 12:17:25.569763 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:17:25.569722    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70b937e_f686_4efd_9950_0aa472f0169b.slice/crio-e37c7c32ed7959a875e762e2294ae6bf7dc10f896a91b550553f61acac416b4b WatchSource:0}: Error finding container e37c7c32ed7959a875e762e2294ae6bf7dc10f896a91b550553f61acac416b4b: Status 404 returned error can't find the container with id e37c7c32ed7959a875e762e2294ae6bf7dc10f896a91b550553f61acac416b4b
Nov 03 12:17:25.572152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.572121    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:17:25.572209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.572187    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:17:25.572241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:25.572216    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:17:26.260228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:26.260191    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" event={"ID":"e70b937e-f686-4efd-9950-0aa472f0169b","Type":"ContainerStarted","Data":"4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"}
Nov 03 12:17:26.260643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:26.260251    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" event={"ID":"e70b937e-f686-4efd-9950-0aa472f0169b","Type":"ContainerStarted","Data":"e37c7c32ed7959a875e762e2294ae6bf7dc10f896a91b550553f61acac416b4b"}
Nov 03 12:17:26.370186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:26.370129    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podStartSLOduration=2.37011112 podStartE2EDuration="2.37011112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:26.367898018 +0000 UTC m=+1587.939481981" watchObservedRunningTime="2026-11-03 12:17:26.37011112 +0000 UTC m=+1587.941695033"
Nov 03 12:17:27.137396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.137362    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:27.143728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.143702    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:27.269322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.269291    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:27.270348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.270328    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:17:27.293131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.293097    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn" podUID="88a7e4df-942e-4652-82f7-55a399173321" containerName="payload-processing" containerID="cri-o://ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a" gracePeriod=25
Nov 03 12:17:27.396548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.396462    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94" podUID="52a978a7-02ca-4a74-896a-899048ec7722" containerName="payload-pre-processing" containerID="cri-o://7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb" gracePeriod=25
Nov 03 12:17:27.616342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.616262    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:17:27.685001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.684912    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm\" (UniqueName: \"kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm\") pod \"88a7e4df-942e-4652-82f7-55a399173321\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") "
Nov 03 12:17:27.685178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.685141    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume\") pod \"88a7e4df-942e-4652-82f7-55a399173321\" (UID: \"88a7e4df-942e-4652-82f7-55a399173321\") "
Nov 03 12:17:27.685948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.685893    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume" pod "88a7e4df-942e-4652-82f7-55a399173321" (UID: "88a7e4df-942e-4652-82f7-55a399173321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:27.687960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.687931    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm" pod "88a7e4df-942e-4652-82f7-55a399173321" (UID: "88a7e4df-942e-4652-82f7-55a399173321"). InnerVolumeSpecName "kube-api-access-nw9dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:27.710725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.710705    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:17:27.786985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.786957    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume\") pod \"52a978a7-02ca-4a74-896a-899048ec7722\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") "
Nov 03 12:17:27.787137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.787086    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f\" (UniqueName: \"kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f\") pod \"52a978a7-02ca-4a74-896a-899048ec7722\" (UID: \"52a978a7-02ca-4a74-896a-899048ec7722\") "
Nov 03 12:17:27.787332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.787308    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume" pod "52a978a7-02ca-4a74-896a-899048ec7722" (UID: "52a978a7-02ca-4a74-896a-899048ec7722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:27.787582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.787562    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9dm\" (UniqueName: \"kubernetes.io/projected/88a7e4df-942e-4652-82f7-55a399173321-kube-api-access-nw9dm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:27.787582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.787580    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a978a7-02ca-4a74-896a-899048ec7722-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:27.787726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.787590    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a7e4df-942e-4652-82f7-55a399173321-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:27.789325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.789306    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f" pod "52a978a7-02ca-4a74-896a-899048ec7722" (UID: "52a978a7-02ca-4a74-896a-899048ec7722"). InnerVolumeSpecName "kube-api-access-sg77f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:27.889428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:27.889392    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg77f\" (UniqueName: \"kubernetes.io/projected/52a978a7-02ca-4a74-896a-899048ec7722-kube-api-access-sg77f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:28.278677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.278641    4546 generic.go:356] "Generic (PLEG): container finished" podID="52a978a7-02ca-4a74-896a-899048ec7722" containerID="7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb" exitCode=0
Nov 03 12:17:28.279143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.278687    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"
Nov 03 12:17:28.279143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.278725    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94" event={"ID":"52a978a7-02ca-4a74-896a-899048ec7722","Type":"ContainerDied","Data":"7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb"}
Nov 03 12:17:28.279143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.278809    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94" event={"ID":"52a978a7-02ca-4a74-896a-899048ec7722","Type":"ContainerDied","Data":"1291bf87f40748dad8baebb575695ade84db368af541fc760b2eb42b6f10e125"}
Nov 03 12:17:28.279143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.278840    4546 scope.go:122] "RemoveContainer" containerID="7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb"
Nov 03 12:17:28.280257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.280236    4546 generic.go:356] "Generic (PLEG): container finished" podID="88a7e4df-942e-4652-82f7-55a399173321" containerID="ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a" exitCode=0
Nov 03 12:17:28.280547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.280280    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"
Nov 03 12:17:28.280547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.280332    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn" event={"ID":"88a7e4df-942e-4652-82f7-55a399173321","Type":"ContainerDied","Data":"ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a"}
Nov 03 12:17:28.280547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.280389    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn" event={"ID":"88a7e4df-942e-4652-82f7-55a399173321","Type":"ContainerDied","Data":"1513a332f2403d61facbd1ae53fd0e8df490636f8d12473be304ba1585cce2ed"}
Nov 03 12:17:28.290975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.290952    4546 scope.go:122] "RemoveContainer" containerID="7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb"
Nov 03 12:17:28.291235 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:28.291218    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb\": container with ID starting with 7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb not found: ID does not exist" containerID="7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb"
Nov 03 12:17:28.291309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.291246    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb"} err="failed to get container status \"7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb\": rpc error: code = NotFound desc = could not find container \"7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb\": container with ID starting with 7e4a92eb4d1995189e8cb312489656963d2addbb61167a88c012bbe3d16967bb not found: ID does not exist"
Nov 03 12:17:28.291309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.291267    4546 scope.go:122] "RemoveContainer" containerID="ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a"
Nov 03 12:17:28.300997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.300807    4546 scope.go:122] "RemoveContainer" containerID="ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a"
Nov 03 12:17:28.301081 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:28.301058    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a\": container with ID starting with ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a not found: ID does not exist" containerID="ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a"
Nov 03 12:17:28.301140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.301092    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a"} err="failed to get container status \"ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a\": rpc error: code = NotFound desc = could not find container \"ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a\": container with ID starting with ecdc44bbfa368d5732befd93a170ffd753f747c7de390643a7797f9b86a23b3a not found: ID does not exist"
Nov 03 12:17:28.432717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.432687    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"]
Nov 03 12:17:28.436796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.436767    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac6071-567b7f75b7-wdx94"]
Nov 03 12:17:28.537803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.537677    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"]
Nov 03 12:17:28.541933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.541904    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac6071-5df8b9c67f-bq9pn"]
Nov 03 12:17:28.583310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.583272    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a978a7-02ca-4a74-896a-899048ec7722" path="/var/lib/kubelet/pods/52a978a7-02ca-4a74-896a-899048ec7722/volumes"
Nov 03 12:17:28.583775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:28.583740    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a7e4df-942e-4652-82f7-55a399173321" path="/var/lib/kubelet/pods/88a7e4df-942e-4652-82f7-55a399173321/volumes"
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.194844    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"]
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.197769    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ba3bf50-sw6rq"]
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.206841    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"]
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.207878    4546 cpu_manager.go:414] "containerMap: removing container" podUID="88a7e4df-942e-4652-82f7-55a399173321" containerName="payload-processing"
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.207896    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a7e4df-942e-4652-82f7-55a399173321" containerName="payload-processing"
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.207912    4546 cpu_manager.go:414] "containerMap: removing container" podUID="52a978a7-02ca-4a74-896a-899048ec7722" containerName="payload-pre-processing"
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.207920    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a978a7-02ca-4a74-896a-899048ec7722" containerName="payload-pre-processing"
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.208068    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a7e4df-942e-4652-82f7-55a399173321" containerName="payload-processing"
Nov 03 12:17:35.210071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.208080    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a978a7-02ca-4a74-896a-899048ec7722" containerName="payload-pre-processing"
Nov 03 12:17:35.214091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.214071    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.215721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.215702    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"]
Nov 03 12:17:35.264348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.264318    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e8d12\""
Nov 03 12:17:35.264528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.264441    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e8d12-dockercfg-v7gkm\""
Nov 03 12:17:35.353552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.353501    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.353837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.353819    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ndb\" (UniqueName: \"kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb\") pod \"payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.455010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.454971    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ndb\" (UniqueName: \"kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb\") pod \"payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.455212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.455085    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.455855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.455836    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.508613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.508574    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ndb\" (UniqueName: \"kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb\") pod \"payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.534573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.534522    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:35.962823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:35.953200    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"]
Nov 03 12:17:36.353508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:36.353470    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq" event={"ID":"91eeb679-7ad2-48a4-b977-f86ee4f4d788","Type":"ContainerStarted","Data":"d7d8b6a647852cb2e49f5689d3b7ea2d6bf6da5cdfa3bcbabd903a3583321742"}
Nov 03 12:17:36.353940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:36.353557    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq" event={"ID":"91eeb679-7ad2-48a4-b977-f86ee4f4d788","Type":"ContainerStarted","Data":"26ba933d8756b9dff1fcb8719aaebe68fe49251d0caa99f47ca2009027c5b2ce"}
Nov 03 12:17:36.353940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:36.353720    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:36.462374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:36.462320    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq" podStartSLOduration=1.462306184 podStartE2EDuration="1.462306184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:17:36.458726131 +0000 UTC m=+1598.030310064" watchObservedRunningTime="2026-11-03 12:17:36.462306184 +0000 UTC m=+1598.033890097"
Nov 03 12:17:36.575433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:36.575402    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4d1c17-5e62-4362-a4f5-d07efbd33367" path="/var/lib/kubelet/pods/ef4d1c17-5e62-4362-a4f5-d07efbd33367/volumes"
Nov 03 12:17:37.655326 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:37.655287    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:17:46.110372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:46.110335    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"]
Nov 03 12:17:46.110892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:46.110526    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy" containerID="cri-o://658214e7761a0fada328df211225beb29522795b1001dd0299671ddb277ad40b" gracePeriod=30
Nov 03 12:17:47.364418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:47.364389    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:17:48.044408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:48.044374    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"]
Nov 03 12:17:48.048326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:48.048305    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a54034b-wc6ms"]
Nov 03 12:17:48.573707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:48.573676    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0857f66d-4857-412a-96e9-3edebc47cf2c" path="/var/lib/kubelet/pods/0857f66d-4857-412a-96e9-3edebc47cf2c/volumes"
Nov 03 12:17:50.044006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:50.043972    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"]
Nov 03 12:17:50.089909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:50.089860    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a51f5c-wtpkf"]
Nov 03 12:17:50.576540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:50.576476    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38f684d-2c50-4c1f-9d3e-11b1644a7c02" path="/var/lib/kubelet/pods/c38f684d-2c50-4c1f-9d3e-11b1644a7c02/volumes"
Nov 03 12:17:51.507089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.506636    4546 generic.go:356] "Generic (PLEG): container finished" podID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerID="658214e7761a0fada328df211225beb29522795b1001dd0299671ddb277ad40b" exitCode=0
Nov 03 12:17:51.507089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.506866    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" event={"ID":"bca94ff8-2b4f-4d80-a494-f6c685a91394","Type":"ContainerDied","Data":"658214e7761a0fada328df211225beb29522795b1001dd0299671ddb277ad40b"}
Nov 03 12:17:51.507089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.506939    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q" event={"ID":"bca94ff8-2b4f-4d80-a494-f6c685a91394","Type":"ContainerDied","Data":"5b24c24e0780008a8dcf4b47340fe3fb2beaf4f8b2edec80850519236e4c47b5"}
Nov 03 12:17:51.507089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.506964    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b24c24e0780008a8dcf4b47340fe3fb2beaf4f8b2edec80850519236e4c47b5"
Nov 03 12:17:51.507089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.506981    4546 scope.go:122] "RemoveContainer" containerID="59efadd11dc50c08ddfaca9de1da1d48d1464abf5d2e6a9a51ab8b349b99e3f2"
Nov 03 12:17:51.518110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.518073    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:17:51.665263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665235    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665329    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665458    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665516    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665559    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:51.665691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665665    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665692    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:51.665861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665824    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665869    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665916    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.665975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665926    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:51.666085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.665996    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket\") pod \"bca94ff8-2b4f-4d80-a494-f6c685a91394\" (UID: \"bca94ff8-2b4f-4d80-a494-f6c685a91394\") "
Nov 03 12:17:51.666189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.666167    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:51.666944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.666444    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:51.667113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.667095    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.667236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.667213    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.667316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.667245    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca94ff8-2b4f-4d80-a494-f6c685a91394-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.667316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.667259    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.667316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.667273    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.668101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.668081    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "kube-api-access-jqg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:51.668184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.668171    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:51.668429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.668408    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:17:51.668541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.668435    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo" pod "bca94ff8-2b4f-4d80-a494-f6c685a91394" (UID: "bca94ff8-2b4f-4d80-a494-f6c685a91394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:17:51.768730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.768697    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.768730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.768722    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.768730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.768732    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqg9c\" (UniqueName: \"kubernetes.io/projected/bca94ff8-2b4f-4d80-a494-f6c685a91394-kube-api-access-jqg9c\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.768988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.768743    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca94ff8-2b4f-4d80-a494-f6c685a91394-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.824158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.824125    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"]
Nov 03 12:17:51.922176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:51.922140    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"]
Nov 03 12:17:52.515651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:52.515622    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"
Nov 03 12:17:52.678480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:52.678451    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"]
Nov 03 12:17:52.682377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:52.682351    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-2lvmk"]
Nov 03 12:17:52.732741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:52.732697    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"]
Nov 03 12:17:52.737281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:52.737256    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-86d7ad-openshift-default-86bbcfc549-wxf2q"]
Nov 03 12:17:54.068004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:54.067975    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"]
Nov 03 12:17:54.171505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:54.171471    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"]
Nov 03 12:17:54.589425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:54.589388    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" path="/var/lib/kubelet/pods/bca94ff8-2b4f-4d80-a494-f6c685a91394/volumes"
Nov 03 12:17:54.589967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:54.589951    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13bd48c-139e-410d-9ffc-95f8c7bc5879" path="/var/lib/kubelet/pods/e13bd48c-139e-410d-9ffc-95f8c7bc5879/volumes"
Nov 03 12:17:56.825324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:56.825278    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp" podUID="5ca31a82-9146-4937-9e94-a56e03c81dc8" containerName="payload-processing" containerID="cri-o://cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce" gracePeriod=25
Nov 03 12:17:56.922599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:56.922561    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc" podUID="68d8fa43-9174-4a27-87a7-4f9db69aca42" containerName="payload-pre-processing" containerID="cri-o://a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100" gracePeriod=25
Nov 03 12:17:57.131859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.131832    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:17:57.220738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.212914    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:17:57.243832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.243795    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7\" (UniqueName: \"kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7\") pod \"5ca31a82-9146-4937-9e94-a56e03c81dc8\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") "
Nov 03 12:17:57.244067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.244027    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume\") pod \"5ca31a82-9146-4937-9e94-a56e03c81dc8\" (UID: \"5ca31a82-9146-4937-9e94-a56e03c81dc8\") "
Nov 03 12:17:57.245066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.245035    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume" pod "5ca31a82-9146-4937-9e94-a56e03c81dc8" (UID: "5ca31a82-9146-4937-9e94-a56e03c81dc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:57.247011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.246985    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7" pod "5ca31a82-9146-4937-9e94-a56e03c81dc8" (UID: "5ca31a82-9146-4937-9e94-a56e03c81dc8"). InnerVolumeSpecName "kube-api-access-5k8g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:57.346342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.346108    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c\" (UniqueName: \"kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c\") pod \"68d8fa43-9174-4a27-87a7-4f9db69aca42\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") "
Nov 03 12:17:57.346734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.346408    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume\") pod \"68d8fa43-9174-4a27-87a7-4f9db69aca42\" (UID: \"68d8fa43-9174-4a27-87a7-4f9db69aca42\") "
Nov 03 12:17:57.346883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.346836    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume" pod "68d8fa43-9174-4a27-87a7-4f9db69aca42" (UID: "68d8fa43-9174-4a27-87a7-4f9db69aca42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:57.347831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.347648    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca31a82-9146-4937-9e94-a56e03c81dc8-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:57.347831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.347677    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d8fa43-9174-4a27-87a7-4f9db69aca42-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:57.347831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.347694    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8g7\" (UniqueName: \"kubernetes.io/projected/5ca31a82-9146-4937-9e94-a56e03c81dc8-kube-api-access-5k8g7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:57.350399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.350365    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c" pod "68d8fa43-9174-4a27-87a7-4f9db69aca42" (UID: "68d8fa43-9174-4a27-87a7-4f9db69aca42"). InnerVolumeSpecName "kube-api-access-9v28c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:57.449195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.449166    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v28c\" (UniqueName: \"kubernetes.io/projected/68d8fa43-9174-4a27-87a7-4f9db69aca42-kube-api-access-9v28c\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:57.592154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.592126    4546 generic.go:356] "Generic (PLEG): container finished" podID="68d8fa43-9174-4a27-87a7-4f9db69aca42" containerID="a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100" exitCode=0
Nov 03 12:17:57.592317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.592164    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"
Nov 03 12:17:57.592317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.592212    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc" event={"ID":"68d8fa43-9174-4a27-87a7-4f9db69aca42","Type":"ContainerDied","Data":"a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100"}
Nov 03 12:17:57.592317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.592261    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc" event={"ID":"68d8fa43-9174-4a27-87a7-4f9db69aca42","Type":"ContainerDied","Data":"fcb1f0d645f3e21b6d2635f8a62af6b837d72efa4490750a27cefe2eb3eba532"}
Nov 03 12:17:57.592317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.592291    4546 scope.go:122] "RemoveContainer" containerID="a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100"
Nov 03 12:17:57.593942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.593922    4546 generic.go:356] "Generic (PLEG): container finished" podID="5ca31a82-9146-4937-9e94-a56e03c81dc8" containerID="cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce" exitCode=0
Nov 03 12:17:57.594055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.593972    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp" event={"ID":"5ca31a82-9146-4937-9e94-a56e03c81dc8","Type":"ContainerDied","Data":"cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce"}
Nov 03 12:17:57.594055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.594011    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp" event={"ID":"5ca31a82-9146-4937-9e94-a56e03c81dc8","Type":"ContainerDied","Data":"685801ede449f6a5b424ec352c369c93399d5960786f9df4c6f26fd9a7ad8595"}
Nov 03 12:17:57.594133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.594112    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"
Nov 03 12:17:57.605158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.605135    4546 scope.go:122] "RemoveContainer" containerID="a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100"
Nov 03 12:17:57.605475 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:57.605451    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100\": container with ID starting with a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100 not found: ID does not exist" containerID="a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100"
Nov 03 12:17:57.605534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.605486    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100"} err="failed to get container status \"a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100\": rpc error: code = NotFound desc = could not find container \"a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100\": container with ID starting with a8c020576349ca0e389dea62078a5c17a945bdeab0a999f2061483d31b8ea100 not found: ID does not exist"
Nov 03 12:17:57.605534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.605512    4546 scope.go:122] "RemoveContainer" containerID="cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce"
Nov 03 12:17:57.617516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.617496    4546 scope.go:122] "RemoveContainer" containerID="cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce"
Nov 03 12:17:57.617784 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:57.617763    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce\": container with ID starting with cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce not found: ID does not exist" containerID="cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce"
Nov 03 12:17:57.617850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.617790    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce"} err="failed to get container status \"cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce\": rpc error: code = NotFound desc = could not find container \"cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce\": container with ID starting with cc757f3ff19a4aaf34e37173290fb9daf3f89338a9d09d9a28e35f16cc6436ce not found: ID does not exist"
Nov 03 12:17:57.747996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.747964    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"]
Nov 03 12:17:57.794675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.794647    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-931523-b9fc8886c-nhcdc"]
Nov 03 12:17:57.855172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.855089    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"]
Nov 03 12:17:57.858817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:57.858793    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-931523-54b99f97bc-5n5pp"]
Nov 03 12:17:58.574527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:58.574496    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca31a82-9146-4937-9e94-a56e03c81dc8" path="/var/lib/kubelet/pods/5ca31a82-9146-4937-9e94-a56e03c81dc8/volumes"
Nov 03 12:17:58.574951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:58.574927    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d8fa43-9174-4a27-87a7-4f9db69aca42" path="/var/lib/kubelet/pods/68d8fa43-9174-4a27-87a7-4f9db69aca42/volumes"
Nov 03 12:17:59.068853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.068794    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg" podUID="02a8de4e-ba60-4c90-8def-64ce7046318e" containerName="payload-processing" containerID="cri-o://71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887" gracePeriod=25
Nov 03 12:17:59.172160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.172113    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6" podUID="66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" containerName="payload-pre-processing" containerID="cri-o://d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f" gracePeriod=25
Nov 03 12:17:59.177857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.177835    4546 scope.go:122] "RemoveContainer" containerID="07e0998839d88e29a5dc15ce4b119e3316bad9972e817c1ae5b074aa2ce7a01c"
Nov 03 12:17:59.245586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.245553    4546 scope.go:122] "RemoveContainer" containerID="1c9f0600fc741859e8b9ad9ad50db927b52c989d07a2933f0e4144ab7ac92e1e"
Nov 03 12:17:59.293237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.293204    4546 scope.go:122] "RemoveContainer" containerID="7de636e91cd1ae7d9ef1fbce92ca733a649c927d10d240cce1abbec582ece773"
Nov 03 12:17:59.305805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.305780    4546 scope.go:122] "RemoveContainer" containerID="3bec065e7cc6b201e752f56ad59a4a27dc759a78c05b678c4bfa5eabac83f795"
Nov 03 12:17:59.325885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.325865    4546 scope.go:122] "RemoveContainer" containerID="0e773aaba3cb59582edd85bee5a0449ca5b125b20d492a0e253715cb18570d3a"
Nov 03 12:17:59.363944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.363921    4546 scope.go:122] "RemoveContainer" containerID="791d027e4339992e79504f00206cb619767472895d6e0285e88640280ceb8406"
Nov 03 12:17:59.364932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.364533    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:17:59.385910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.384492    4546 scope.go:122] "RemoveContainer" containerID="b5d370ce278f380e94993ef13c1ac8847535a7f9941eeda838378e31d54ddbf4"
Nov 03 12:17:59.395459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.395430    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6\" (UniqueName: \"kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6\") pod \"02a8de4e-ba60-4c90-8def-64ce7046318e\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") "
Nov 03 12:17:59.395642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.395513    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume\") pod \"02a8de4e-ba60-4c90-8def-64ce7046318e\" (UID: \"02a8de4e-ba60-4c90-8def-64ce7046318e\") "
Nov 03 12:17:59.396529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.396492    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume" pod "02a8de4e-ba60-4c90-8def-64ce7046318e" (UID: "02a8de4e-ba60-4c90-8def-64ce7046318e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:59.398577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.398552    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6" pod "02a8de4e-ba60-4c90-8def-64ce7046318e" (UID: "02a8de4e-ba60-4c90-8def-64ce7046318e"). InnerVolumeSpecName "kube-api-access-4fpj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:59.462995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.462902    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:17:59.497688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.497643    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume\") pod \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") "
Nov 03 12:17:59.497889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.497867    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt\" (UniqueName: \"kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt\") pod \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\" (UID: \"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8\") "
Nov 03 12:17:59.498542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.498517    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpj6\" (UniqueName: \"kubernetes.io/projected/02a8de4e-ba60-4c90-8def-64ce7046318e-kube-api-access-4fpj6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.498657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.498547    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a8de4e-ba60-4c90-8def-64ce7046318e-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.499227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.499200    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume" pod "66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" (UID: "66d3dd9c-5d87-42a0-90d8-2248e9e5dca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:59.501418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.501395    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt" pod "66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" (UID: "66d3dd9c-5d87-42a0-90d8-2248e9e5dca8"). InnerVolumeSpecName "kube-api-access-wc4jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:59.600690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.600588    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4jt\" (UniqueName: \"kubernetes.io/projected/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-kube-api-access-wc4jt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.600690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.600624    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:59.615527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.615494    4546 generic.go:356] "Generic (PLEG): container finished" podID="66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" containerID="d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f" exitCode=0
Nov 03 12:17:59.615787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.615549    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"
Nov 03 12:17:59.615787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.615581    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6" event={"ID":"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8","Type":"ContainerDied","Data":"d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f"}
Nov 03 12:17:59.615787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.615663    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6" event={"ID":"66d3dd9c-5d87-42a0-90d8-2248e9e5dca8","Type":"ContainerDied","Data":"a493589549d2e590b361b7d849f4e5bdb0cd4ae193c67a891cf81d9a151b5ea9"}
Nov 03 12:17:59.615787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.615722    4546 scope.go:122] "RemoveContainer" containerID="d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f"
Nov 03 12:17:59.621573 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.621529    4546 generic.go:356] "Generic (PLEG): container finished" podID="02a8de4e-ba60-4c90-8def-64ce7046318e" containerID="71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887" exitCode=0
Nov 03 12:17:59.621834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.621810    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg" event={"ID":"02a8de4e-ba60-4c90-8def-64ce7046318e","Type":"ContainerDied","Data":"71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887"}
Nov 03 12:17:59.621926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.621863    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg" event={"ID":"02a8de4e-ba60-4c90-8def-64ce7046318e","Type":"ContainerDied","Data":"84324242d1d6f0189f26ac4a5e2d68d92071885e5567eba17a21f7e540175bfd"}
Nov 03 12:17:59.621986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.621946    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"
Nov 03 12:17:59.634187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.634170    4546 scope.go:122] "RemoveContainer" containerID="d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f"
Nov 03 12:17:59.634469 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:59.634454    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f\": container with ID starting with d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f not found: ID does not exist" containerID="d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f"
Nov 03 12:17:59.634528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.634475    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f"} err="failed to get container status \"d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f\": rpc error: code = NotFound desc = could not find container \"d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f\": container with ID starting with d4d4ecca2a2b28be314b72a384ddc19f1bb42648b4b5c50400e9bd242389ce8f not found: ID does not exist"
Nov 03 12:17:59.634528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.634493    4546 scope.go:122] "RemoveContainer" containerID="71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887"
Nov 03 12:17:59.644584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.644559    4546 scope.go:122] "RemoveContainer" containerID="71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887"
Nov 03 12:17:59.644906 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:17:59.644882    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887\": container with ID starting with 71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887 not found: ID does not exist" containerID="71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887"
Nov 03 12:17:59.644971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.644917    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887"} err="failed to get container status \"71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887\": rpc error: code = NotFound desc = could not find container \"71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887\": container with ID starting with 71077bca2cab538787011011ff17b3df663e81f65ed70273a83fca4e50f8f887 not found: ID does not exist"
Nov 03 12:17:59.770504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.770472    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"]
Nov 03 12:17:59.774191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.774165    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-913a17-67f4745fc-hsxkg"]
Nov 03 12:17:59.874900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.874812    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"]
Nov 03 12:17:59.876685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:17:59.876661    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-913a17-7c9b8dd967-blnd6"]
Nov 03 12:18:00.574838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:00.574801    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a8de4e-ba60-4c90-8def-64ce7046318e" path="/var/lib/kubelet/pods/02a8de4e-ba60-4c90-8def-64ce7046318e/volumes"
Nov 03 12:18:00.575435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:00.575413    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" path="/var/lib/kubelet/pods/66d3dd9c-5d87-42a0-90d8-2248e9e5dca8/volumes"
Nov 03 12:18:02.860577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.860539    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"]
Nov 03 12:18:02.861281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861264    4546 cpu_manager.go:414] "containerMap: removing container" podUID="02a8de4e-ba60-4c90-8def-64ce7046318e" containerName="payload-processing"
Nov 03 12:18:02.861340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861282    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a8de4e-ba60-4c90-8def-64ce7046318e" containerName="payload-processing"
Nov 03 12:18:02.861340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861299    4546 cpu_manager.go:414] "containerMap: removing container" podUID="68d8fa43-9174-4a27-87a7-4f9db69aca42" containerName="payload-pre-processing"
Nov 03 12:18:02.861340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861304    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d8fa43-9174-4a27-87a7-4f9db69aca42" containerName="payload-pre-processing"
Nov 03 12:18:02.861340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861314    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5ca31a82-9146-4937-9e94-a56e03c81dc8" containerName="payload-processing"
Nov 03 12:18:02.861340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861319    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca31a82-9146-4937-9e94-a56e03c81dc8" containerName="payload-processing"
Nov 03 12:18:02.861340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861341    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861347    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861366    4546 cpu_manager.go:414] "containerMap: removing container" podUID="66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" containerName="payload-pre-processing"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861371    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" containerName="payload-pre-processing"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861498    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d8fa43-9174-4a27-87a7-4f9db69aca42" containerName="payload-pre-processing"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861509    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca31a82-9146-4937-9e94-a56e03c81dc8" containerName="payload-processing"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861519    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861528    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d3dd9c-5d87-42a0-90d8-2248e9e5dca8" containerName="payload-pre-processing"
Nov 03 12:18:02.861533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861535    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy"
Nov 03 12:18:02.861917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.861542    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a8de4e-ba60-4c90-8def-64ce7046318e" containerName="payload-processing"
Nov 03 12:18:02.866808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.866790    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:02.916107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.916080    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-69fd8043\""
Nov 03 12:18:02.916107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.916088    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-69fd8043-dockercfg-vlxp6\""
Nov 03 12:18:02.921222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.921189    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:02.921402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.921381    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:02.968410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:02.968361    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"]
Nov 03 12:18:03.022894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.022861    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:03.023126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.023059    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:03.023763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.023735    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:03.083770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.082427    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx\") pod \"payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:03.190189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.190101    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:03.585565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.585533    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"]
Nov 03 12:18:03.586058 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:18:03.586028    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd17a3b_6147_4474_ad8b_84079199bacb.slice/crio-6e7a7f9f98493883ace8072686687c6a5b80b5f20291b38ff82eb8faa6b41550 WatchSource:0}: Error finding container 6e7a7f9f98493883ace8072686687c6a5b80b5f20291b38ff82eb8faa6b41550: Status 404 returned error can't find the container with id 6e7a7f9f98493883ace8072686687c6a5b80b5f20291b38ff82eb8faa6b41550
Nov 03 12:18:03.657742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:03.657713    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb" event={"ID":"bfd17a3b-6147-4474-ad8b-84079199bacb","Type":"ContainerStarted","Data":"6e7a7f9f98493883ace8072686687c6a5b80b5f20291b38ff82eb8faa6b41550"}
Nov 03 12:18:04.667243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:04.667201    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb" event={"ID":"bfd17a3b-6147-4474-ad8b-84079199bacb","Type":"ContainerStarted","Data":"f7aaf62148cb15ebc24606d6c091e95f91ec456f9874eccd79fa83d77a347cd9"}
Nov 03 12:18:04.667631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:04.667322    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:04.773030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:04.772986    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb" podStartSLOduration=2.772973225 podStartE2EDuration="2.772973225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:04.772107058 +0000 UTC m=+1626.343690981" watchObservedRunningTime="2026-11-03 12:18:04.772973225 +0000 UTC m=+1626.344557137"
Nov 03 12:18:07.652872 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:18:07.652834    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:18:15.677733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:15.677704    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:18:20.046918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:20.046889    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"]
Nov 03 12:18:20.092600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:20.092571    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eeae3cb-z7jpg"]
Nov 03 12:18:20.575303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:20.575272    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098d9b59-e9bc-44e3-9382-de1f3e4e2ef3" path="/var/lib/kubelet/pods/098d9b59-e9bc-44e3-9382-de1f3e4e2ef3/volumes"
Nov 03 12:18:27.046949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:27.046904    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"]
Nov 03 12:18:27.089211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:27.089182    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da65ecf-6gbzn"]
Nov 03 12:18:28.576193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:28.576165    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d1aad2-3cc7-4c05-9bec-7468761ad6c8" path="/var/lib/kubelet/pods/e8d1aad2-3cc7-4c05-9bec-7468761ad6c8/volumes"
Nov 03 12:18:31.045454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:31.045426    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"]
Nov 03 12:18:31.091715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:31.091689    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe07748b-bk5nq"]
Nov 03 12:18:32.044610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:32.044576    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"]
Nov 03 12:18:32.090276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:32.090244    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d32f8f70-22n82"]
Nov 03 12:18:32.573472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:32.573442    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a680f5-ed92-4b24-8d1f-354556aa7ad5" path="/var/lib/kubelet/pods/01a680f5-ed92-4b24-8d1f-354556aa7ad5/volumes"
Nov 03 12:18:32.573955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:32.573942    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693040f0-dc0f-4165-87cb-ebb6e282b477" path="/var/lib/kubelet/pods/693040f0-dc0f-4165-87cb-ebb6e282b477/volumes"
Nov 03 12:18:35.054139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:35.054094    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"]
Nov 03 12:18:35.101089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:35.101038    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f5629f5-chh7j"]
Nov 03 12:18:36.233941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:36.233903    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"]
Nov 03 12:18:36.234657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:36.234605    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" containerID="cri-o://e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc" gracePeriod=30
Nov 03 12:18:36.574088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:36.574015    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f85ddc4-d217-40d4-8e4b-5283ccc27808" path="/var/lib/kubelet/pods/4f85ddc4-d217-40d4-8e4b-5283ccc27808/volumes"
Nov 03 12:18:37.663275 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:18:37.663184    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:18:39.543355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:39.543318    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"]
Nov 03 12:18:39.557960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:39.557920    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy" containerID="cri-o://b42f12542c3234e783f4e122f24da5a63588b634156277aa4bcc4c6852944045" gracePeriod=30
Nov 03 12:18:41.396912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.396856    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 12:18:41.397440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.396946    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 12:18:41.511266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.511203    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:18:41.592240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592199    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592269    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592316    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592426    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592495    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592530    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592631    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592765 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592709    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592796    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo\") pod \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\" (UID: \"2d16bfbf-11ea-4e78-8177-7b5771944e3c\") "
Nov 03 12:18:41.592907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.592852    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.594827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.593480    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:41.594827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.593652    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.594827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.593672    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.594827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.594015    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.594827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.594695    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.594827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.594796    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.599311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.599275    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.599574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.599542    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:41.600230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.600197    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "kube-api-access-qxbl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:41.605893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.605861    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo" pod "2d16bfbf-11ea-4e78-8177-7b5771944e3c" (UID: "2d16bfbf-11ea-4e78-8177-7b5771944e3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:41.617277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.612650    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"]
Nov 03 12:18:41.617277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.612941    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy" containerID="cri-o://0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb" gracePeriod=30
Nov 03 12:18:41.695692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695653    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxbl9\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-kube-api-access-qxbl9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.695692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695689    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.695987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695702    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.695987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695715    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.695987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695726    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.695987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695740    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.695987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.695787    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16bfbf-11ea-4e78-8177-7b5771944e3c-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.974778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.974729    4546 generic.go:356] "Generic (PLEG): container finished" podID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerID="e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc" exitCode=0
Nov 03 12:18:41.974963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.974783    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" event={"ID":"2d16bfbf-11ea-4e78-8177-7b5771944e3c","Type":"ContainerDied","Data":"e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc"}
Nov 03 12:18:41.974963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.974814    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"
Nov 03 12:18:41.974963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.974854    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q" event={"ID":"2d16bfbf-11ea-4e78-8177-7b5771944e3c","Type":"ContainerDied","Data":"275b5233a053dd8ff41edc97805187c0e051a12c1147018e9d102dc76fb0deb3"}
Nov 03 12:18:41.974963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.974885    4546 scope.go:122] "RemoveContainer" containerID="e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc"
Nov 03 12:18:41.988327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.988308    4546 scope.go:122] "RemoveContainer" containerID="e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc"
Nov 03 12:18:41.988687 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:18:41.988666    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc\": container with ID starting with e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc not found: ID does not exist" containerID="e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc"
Nov 03 12:18:41.988820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:41.988702    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc"} err="failed to get container status \"e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc\": rpc error: code = NotFound desc = could not find container \"e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc\": container with ID starting with e88cb81a10c4775dd2313f9e9227e1658cdc363d1513a640d6692e9d05a073bc not found: ID does not exist"
Nov 03 12:18:42.128824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:42.128783    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"]
Nov 03 12:18:42.174462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:42.174404    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-cqw9q"]
Nov 03 12:18:42.575057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:42.575025    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" path="/var/lib/kubelet/pods/2d16bfbf-11ea-4e78-8177-7b5771944e3c/volumes"
Nov 03 12:18:43.311887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:43.311855    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:18:43.312083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:43.311924    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:18:44.857012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.856970    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"]
Nov 03 12:18:44.861066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.858029    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy"
Nov 03 12:18:44.861066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.858051    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca94ff8-2b4f-4d80-a494-f6c685a91394" containerName="istio-proxy"
Nov 03 12:18:44.861066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.858086    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy"
Nov 03 12:18:44.861066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.858094    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy"
Nov 03 12:18:44.861066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.858226    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d16bfbf-11ea-4e78-8177-7b5771944e3c" containerName="istio-proxy"
Nov 03 12:18:44.867423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.867400    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962320    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962385    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962460    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962480    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962501    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962520    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962564    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962705    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mchn\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962819    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:44.966071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.962950    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"]
Nov 03 12:18:44.982068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:44.982021    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-deb666-openshift-default-dockercfg-rhz5p\""
Nov 03 12:18:45.026078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.022855    4546 generic.go:356] "Generic (PLEG): container finished" podID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerID="b42f12542c3234e783f4e122f24da5a63588b634156277aa4bcc4c6852944045" exitCode=0
Nov 03 12:18:45.026078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.023059    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" event={"ID":"89bbea76-9808-42e8-af21-8b2c11cef1c6","Type":"ContainerDied","Data":"b42f12542c3234e783f4e122f24da5a63588b634156277aa4bcc4c6852944045"}
Nov 03 12:18:45.058482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.058320    4546 generic.go:356] "Generic (PLEG): container finished" podID="e70b937e-f686-4efd-9950-0aa472f0169b" containerID="4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4" exitCode=0
Nov 03 12:18:45.058883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.058494    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" event={"ID":"e70b937e-f686-4efd-9950-0aa472f0169b","Type":"ContainerDied","Data":"4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"}
Nov 03 12:18:45.059546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.059512    4546 scope.go:122] "RemoveContainer" containerID="4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"
Nov 03 12:18:45.065130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.061670    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:45.065130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.061907    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:45.065130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.061998    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.065781    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mchn\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.065867    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.065942    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.065992    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.066053    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.066076    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.066101    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.066127    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.071827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.066172    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.076456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.076279    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.077346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.076812    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.080847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.080537    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.084955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.083650    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.084955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.083962    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.084955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.084192    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.097773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.095164    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.124161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.124135    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:18:45.127091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.127065    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.127861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.127835    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mchn\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn\") pod \"e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") " pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.167111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167087    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167125    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167170    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167198    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167252    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167357    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:45.167675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167453    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167485    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167601    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.167675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.167644    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs\") pod \"89bbea76-9808-42e8-af21-8b2c11cef1c6\" (UID: \"89bbea76-9808-42e8-af21-8b2c11cef1c6\") "
Nov 03 12:18:45.171147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.168049    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:45.171147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.168326    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.171147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.168345    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.171147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.169531    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:45.171147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.169962    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:45.171147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.170719    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:45.177419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.172926    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:45.177419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.172960    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:45.177419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.173054    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:45.177419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.174882    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29" pod "89bbea76-9808-42e8-af21-8b2c11cef1c6" (UID: "89bbea76-9808-42e8-af21-8b2c11cef1c6"). InnerVolumeSpecName "kube-api-access-vld29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:45.268637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.267410    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:45.296355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296280    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.296355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296310    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bbea76-9808-42e8-af21-8b2c11cef1c6-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.296355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296319    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.296355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296327    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.296355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296341    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bbea76-9808-42e8-af21-8b2c11cef1c6-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.296630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296384    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vld29\" (UniqueName: \"kubernetes.io/projected/89bbea76-9808-42e8-af21-8b2c11cef1c6-kube-api-access-vld29\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.296630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.296422    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bbea76-9808-42e8-af21-8b2c11cef1c6-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:45.737542 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:18:45.737498    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52bbfaa_1be7_40e7_9caa_997651d7155c.slice/crio-4f7079f3bca5f68bba34fcb1af9b56499b8e37362afdba41570df2eb46f2d0f5 WatchSource:0}: Error finding container 4f7079f3bca5f68bba34fcb1af9b56499b8e37362afdba41570df2eb46f2d0f5: Status 404 returned error can't find the container with id 4f7079f3bca5f68bba34fcb1af9b56499b8e37362afdba41570df2eb46f2d0f5
Nov 03 12:18:45.741231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.741192    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:45.741346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.741283    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:45.741346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.741321    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:45.741743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:45.741720    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"]
Nov 03 12:18:46.068439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.068355    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"
Nov 03 12:18:46.068439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.068371    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l" event={"ID":"89bbea76-9808-42e8-af21-8b2c11cef1c6","Type":"ContainerDied","Data":"81978602f988623fc26df75a44b4710a06b6111395301ae13db0663f69e2ae4b"}
Nov 03 12:18:46.068439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.068432    4546 scope.go:122] "RemoveContainer" containerID="b42f12542c3234e783f4e122f24da5a63588b634156277aa4bcc4c6852944045"
Nov 03 12:18:46.071222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.071197    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" event={"ID":"e70b937e-f686-4efd-9950-0aa472f0169b","Type":"ContainerStarted","Data":"40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443"}
Nov 03 12:18:46.073175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.073153    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" event={"ID":"d52bbfaa-1be7-40e7-9caa-997651d7155c","Type":"ContainerStarted","Data":"07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"}
Nov 03 12:18:46.073282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.073210    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" event={"ID":"d52bbfaa-1be7-40e7-9caa-997651d7155c","Type":"ContainerStarted","Data":"4f7079f3bca5f68bba34fcb1af9b56499b8e37362afdba41570df2eb46f2d0f5"}
Nov 03 12:18:46.137390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.137355    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:18:46.137682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.137655    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 12:18:46.137797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.137712    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 12:18:46.232132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.232073    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podStartSLOduration=2.232054077 podStartE2EDuration="2.232054077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:46.230296199 +0000 UTC m=+1667.801880138" watchObservedRunningTime="2026-11-03 12:18:46.232054077 +0000 UTC m=+1667.803637990"
Nov 03 12:18:46.267984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.267953    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:46.273382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.273357    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:46.336910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.336793    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"]
Nov 03 12:18:46.340152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.340126    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-fz77l"]
Nov 03 12:18:46.573642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.573612    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" path="/var/lib/kubelet/pods/89bbea76-9808-42e8-af21-8b2c11cef1c6/volumes"
Nov 03 12:18:46.855982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.855714    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:18:46.943377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943299    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943392    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943476    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943527    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943571    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:46.943714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943638    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943792    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943844    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943880    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.943938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943918    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket\") pod \"a995bae5-79c9-4200-921e-3940efb438d0\" (UID: \"a995bae5-79c9-4200-921e-3940efb438d0\") "
Nov 03 12:18:46.944032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943935    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:46.944032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.943944    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:46.944310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944222    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:46.944416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944344    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:46.944634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944614    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:46.944690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944643    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:46.944690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944658    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:46.944690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944672    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a995bae5-79c9-4200-921e-3940efb438d0-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:46.944690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.944687    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:46.946042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.946019    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:46.946303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.946282    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:46.946578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.946559    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:46.946627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:46.946580    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd" pod "a995bae5-79c9-4200-921e-3940efb438d0" (UID: "a995bae5-79c9-4200-921e-3940efb438d0"). InnerVolumeSpecName "kube-api-access-dhsvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:47.049707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.046779    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsvd\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-kube-api-access-dhsvd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:47.049707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.046814    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a995bae5-79c9-4200-921e-3940efb438d0-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:47.049707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.046829    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a995bae5-79c9-4200-921e-3940efb438d0-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:47.049707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.046844    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a995bae5-79c9-4200-921e-3940efb438d0-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:47.083473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.083443    4546 generic.go:356] "Generic (PLEG): container finished" podID="a995bae5-79c9-4200-921e-3940efb438d0" containerID="0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb" exitCode=0
Nov 03 12:18:47.083986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.083497    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"
Nov 03 12:18:47.083986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.083524    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" event={"ID":"a995bae5-79c9-4200-921e-3940efb438d0","Type":"ContainerDied","Data":"0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb"}
Nov 03 12:18:47.083986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.083619    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp" event={"ID":"a995bae5-79c9-4200-921e-3940efb438d0","Type":"ContainerDied","Data":"351af2ec18fe5eb770bc00c885118b1784a856ff99266bb5e91914bb5c5bfcb0"}
Nov 03 12:18:47.083986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.083654    4546 scope.go:122] "RemoveContainer" containerID="0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb"
Nov 03 12:18:47.084211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.084187    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:47.085318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.085302    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:18:47.095663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.095642    4546 scope.go:122] "RemoveContainer" containerID="c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"
Nov 03 12:18:47.105058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.105042    4546 scope.go:122] "RemoveContainer" containerID="0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb"
Nov 03 12:18:47.105302 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:18:47.105283    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb\": container with ID starting with 0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb not found: ID does not exist" containerID="0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb"
Nov 03 12:18:47.105348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.105311    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb"} err="failed to get container status \"0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb\": rpc error: code = NotFound desc = could not find container \"0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb\": container with ID starting with 0aade1d00ae20aaa32718a8854f00e542cb0141487cf9ab76b6d544883d405eb not found: ID does not exist"
Nov 03 12:18:47.105348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.105332    4546 scope.go:122] "RemoveContainer" containerID="c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"
Nov 03 12:18:47.105584 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:18:47.105566    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded\": container with ID starting with c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded not found: ID does not exist" containerID="c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"
Nov 03 12:18:47.105642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.105590    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded"} err="failed to get container status \"c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded\": rpc error: code = NotFound desc = could not find container \"c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded\": container with ID starting with c5cce4ebf3fa1f5cb8ee24a2b9788788e7b87a1ee2bbebf6c3bd49e185b02ded not found: ID does not exist"
Nov 03 12:18:47.137541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.137447    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 12:18:47.137693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.137544    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 12:18:47.237822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.237642    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"]
Nov 03 12:18:47.241892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:47.241798    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-913a17-openshift-default-578fd4c75-8qcsp"]
Nov 03 12:18:48.137773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:48.137715    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 12:18:48.138164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:48.137818    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 12:18:48.602328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:48.602281    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a995bae5-79c9-4200-921e-3940efb438d0" path="/var/lib/kubelet/pods/a995bae5-79c9-4200-921e-3940efb438d0/volumes"
Nov 03 12:18:49.136862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:49.136829    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 12:18:49.137054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:49.136890    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 12:18:50.046018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.045984    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"]
Nov 03 12:18:50.091692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.091648    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc42a51-dzx5v"]
Nov 03 12:18:50.137338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.137304    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 12:18:50.137545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.137363    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 12:18:50.472191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472162    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"]
Nov 03 12:18:50.472838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472816    4546 cpu_manager.go:414] "containerMap: removing container" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy"
Nov 03 12:18:50.472901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472839    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy"
Nov 03 12:18:50.472901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472868    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy"
Nov 03 12:18:50.472901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472873    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy"
Nov 03 12:18:50.472901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472883    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy"
Nov 03 12:18:50.472901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472887    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy"
Nov 03 12:18:50.473056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472978    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy"
Nov 03 12:18:50.473056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472985    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a995bae5-79c9-4200-921e-3940efb438d0" containerName="istio-proxy"
Nov 03 12:18:50.473056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.472995    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bbea76-9808-42e8-af21-8b2c11cef1c6" containerName="istio-proxy"
Nov 03 12:18:50.477698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.477680    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.585582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.565329    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1e151c-openshift-default-dockercfg-5tx2q\""
Nov 03 12:18:50.779427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.778149    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5832fc-415a-40c6-a521-33a24588a0fd" path="/var/lib/kubelet/pods/8f5832fc-415a-40c6-a521-33a24588a0fd/volumes"
Nov 03 12:18:50.779427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.778615    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"]
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793267    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793374    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxkz\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793416    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793451    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793522    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793558    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793589    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793767    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.794128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.793797    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.894894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.894867    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.894902    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.894965    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.894990    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895012    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895110    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895126    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895176    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895260    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxkz\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895317    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895449    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.895852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.895832    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.896054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.896034    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.896054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.896046    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.897628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.897605    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.897971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.897952    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.949069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.948913    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:50.949069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:50.949027    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxkz\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz\") pod \"e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") " pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:51.091584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.091544    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:51.137564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.137530    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 12:18:51.137725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.137593    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 12:18:51.525919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.525890    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"]
Nov 03 12:18:51.526122 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:18:51.526102    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1caa903_f2f8_4c8f_a8e2_7550ec5f16a6.slice/crio-4a40c93d21a9515da65eec0a473ba5fe92167bf030d0fecb06e96e267fead0c0 WatchSource:0}: Error finding container 4a40c93d21a9515da65eec0a473ba5fe92167bf030d0fecb06e96e267fead0c0: Status 404 returned error can't find the container with id 4a40c93d21a9515da65eec0a473ba5fe92167bf030d0fecb06e96e267fead0c0
Nov 03 12:18:51.528695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.528665    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:51.528799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.528730    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:51.528799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:51.528778    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:18:52.125138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:52.125102    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" event={"ID":"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6","Type":"ContainerStarted","Data":"a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"}
Nov 03 12:18:52.125545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:52.125162    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" event={"ID":"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6","Type":"ContainerStarted","Data":"4a40c93d21a9515da65eec0a473ba5fe92167bf030d0fecb06e96e267fead0c0"}
Nov 03 12:18:52.141421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:52.141399    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:18:52.141678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:52.141658    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:18:52.142248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:52.142231    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:18:52.235735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:52.235686    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podStartSLOduration=2.2356726670000002 podStartE2EDuration="2.235672667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:52.233881959 +0000 UTC m=+1673.805465882" watchObservedRunningTime="2026-11-03 12:18:52.235672667 +0000 UTC m=+1673.807256577"
Nov 03 12:18:53.001647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.001609    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"]
Nov 03 12:18:53.018800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.009134    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.073211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.073175    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-deb666\""
Nov 03 12:18:53.073412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.073280    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-deb666-dockercfg-vnpl9\""
Nov 03 12:18:53.078472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.078435    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"]
Nov 03 12:18:53.092127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.092096    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:53.108154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.108118    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:53.162771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.162691    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hld2r\" (UniqueName: \"kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r\") pod \"payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.162771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.162775    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.210890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.210849    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"]
Nov 03 12:18:53.216287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.216248    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:53.216565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.216549    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:18:53.216823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.216799    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.237887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.237859    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"]
Nov 03 12:18:53.264816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.264367    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hld2r\" (UniqueName: \"kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r\") pod \"payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.264816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.264425    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.264816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.264592    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5cv\" (UniqueName: \"kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv\") pod \"payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.266548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.264705    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.267690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.267545    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.319900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.319868    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hld2r\" (UniqueName: \"kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r\") pod \"payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.334476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.334449    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:53.367109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.367074    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5cv\" (UniqueName: \"kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv\") pod \"payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.367239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.367190    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.367841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.367822    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.420222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.420188    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5cv\" (UniqueName: \"kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv\") pod \"payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.555889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.555806    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:53.703999 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:18:53.703969    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe83552_cbbd_44bd_a3fc_b237f8f486c3.slice/crio-b105730de19d3a73effcc6329d91ffa97b833a1aff81fab53b7c79da551c9e53 WatchSource:0}: Error finding container b105730de19d3a73effcc6329d91ffa97b833a1aff81fab53b7c79da551c9e53: Status 404 returned error can't find the container with id b105730de19d3a73effcc6329d91ffa97b833a1aff81fab53b7c79da551c9e53
Nov 03 12:18:53.704532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.704502    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"]
Nov 03 12:18:53.931923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:53.931890    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"]
Nov 03 12:18:53.933480 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:18:53.933449    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08543efc_bb86_4e93_bd36_57f09f0f02af.slice/crio-a72ed71301b2bae0bc38601e5c886e9331df1267585011b13d7ea015f5b63f30 WatchSource:0}: Error finding container a72ed71301b2bae0bc38601e5c886e9331df1267585011b13d7ea015f5b63f30: Status 404 returned error can't find the container with id a72ed71301b2bae0bc38601e5c886e9331df1267585011b13d7ea015f5b63f30
Nov 03 12:18:54.219566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.219521    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l" event={"ID":"ebe83552-cbbd-44bd-a3fc-b237f8f486c3","Type":"ContainerStarted","Data":"c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989"}
Nov 03 12:18:54.220000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.219585    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:18:54.220000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.219605    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l" event={"ID":"ebe83552-cbbd-44bd-a3fc-b237f8f486c3","Type":"ContainerStarted","Data":"b105730de19d3a73effcc6329d91ffa97b833a1aff81fab53b7c79da551c9e53"}
Nov 03 12:18:54.221161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.221136    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4" event={"ID":"08543efc-bb86-4e93-bd36-57f09f0f02af","Type":"ContainerStarted","Data":"73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb"}
Nov 03 12:18:54.221302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.221184    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:18:54.221302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.221201    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4" event={"ID":"08543efc-bb86-4e93-bd36-57f09f0f02af","Type":"ContainerStarted","Data":"a72ed71301b2bae0bc38601e5c886e9331df1267585011b13d7ea015f5b63f30"}
Nov 03 12:18:54.327946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.327901    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l" podStartSLOduration=2.327888677 podStartE2EDuration="2.327888677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:54.325640601 +0000 UTC m=+1675.897224512" watchObservedRunningTime="2026-11-03 12:18:54.327888677 +0000 UTC m=+1675.899472585"
Nov 03 12:18:54.432392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:54.432345    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4" podStartSLOduration=1.432334083 podStartE2EDuration="1.432334083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:54.430178659 +0000 UTC m=+1676.001762587" watchObservedRunningTime="2026-11-03 12:18:54.432334083 +0000 UTC m=+1676.003917994"
Nov 03 12:18:58.689513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.689475    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"]
Nov 03 12:18:58.699273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.699244    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:58.748304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.748265    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"]
Nov 03 12:18:58.780769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.777137    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1e151c\""
Nov 03 12:18:58.780769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.779635    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1e151c-dockercfg-th58v\""
Nov 03 12:18:58.782502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.782476    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5rr\" (UniqueName: \"kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr\") pod \"payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:58.782803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.782739    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:58.885256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.885227    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5rr\" (UniqueName: \"kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr\") pod \"payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:58.885419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.885263    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:58.885988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.885971    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:58.982769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:58.968707    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5rr\" (UniqueName: \"kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr\") pod \"payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:59.032771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.026804    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:18:59.456397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.456368    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"]
Nov 03 12:18:59.489243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.489217    4546 scope.go:122] "RemoveContainer" containerID="aaace34c9e82cb01f3f0e129ec41b99e010b759f8f8f1b9a18c806a2174bd673"
Nov 03 12:18:59.503539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.503509    4546 scope.go:122] "RemoveContainer" containerID="79fdf92c7a7ed3e8fb991b127024a1907c47280fd102d6cd67b0e8297270d403"
Nov 03 12:18:59.543679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.543651    4546 scope.go:122] "RemoveContainer" containerID="eee62bcf2cd7187ed387a7c3ba56dc054792942435fe8fa7d8b94cdc9c543543"
Nov 03 12:18:59.557966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.557944    4546 scope.go:122] "RemoveContainer" containerID="4e30fc2ace23e5bedc621f93a13956c579b59c7c8a90d09a6ea3013a8b57ccb9"
Nov 03 12:18:59.568409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.568389    4546 scope.go:122] "RemoveContainer" containerID="5de66188075c249b6cbf38c160024dbcbfa7e2a2c56ef11a2c84893f317c6ffe"
Nov 03 12:18:59.577361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.577345    4546 scope.go:122] "RemoveContainer" containerID="a0d35d96ef27a9f33336166ed2abb3160e38961f2e402088b23efad20287c2a8"
Nov 03 12:18:59.586191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:18:59.586172    4546 scope.go:122] "RemoveContainer" containerID="30064496eaa97e5b3a538047ab996a98da4b47cfc80c13528c5e8456f5b86615"
Nov 03 12:19:00.337218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:00.337184    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8" event={"ID":"f702f631-1e55-4bb0-86ff-029906263672","Type":"ContainerStarted","Data":"95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7"}
Nov 03 12:19:00.337687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:00.337294    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:19:00.337687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:00.337336    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8" event={"ID":"f702f631-1e55-4bb0-86ff-029906263672","Type":"ContainerStarted","Data":"799fb314095d43691d72bcdaf4f01c6c185beb2e37a4004553f361d4f62074b1"}
Nov 03 12:19:00.452134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:00.452059    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8" podStartSLOduration=2.45204147 podStartE2EDuration="2.45204147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:00.451070383 +0000 UTC m=+1682.022654294" watchObservedRunningTime="2026-11-03 12:19:00.45204147 +0000 UTC m=+1682.023625381"
Nov 03 12:19:05.264498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:05.264465    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:19:05.265014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:05.264550    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:19:07.288489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:07.288460    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"]
Nov 03 12:19:07.651218 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:07.651134    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:19:09.460466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.460437    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"]
Nov 03 12:19:09.461005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.460611    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" containerID="cri-o://2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd" gracePeriod=30
Nov 03 12:19:09.507517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.507481    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"]
Nov 03 12:19:09.508174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.507738    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" containerID="cri-o://aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614" gracePeriod=30
Nov 03 12:19:09.520894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.520855    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"]
Nov 03 12:19:09.521347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.521305    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" containerID="cri-o://c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a" gracePeriod=30
Nov 03 12:19:09.561285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.561249    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"]
Nov 03 12:19:09.561699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:09.561668    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" containerID="cri-o://3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304" gracePeriod=30
Nov 03 12:19:11.013100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.013062    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:11.013579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.013120    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:11.036014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.035988    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:11.036131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.036034    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:11.347853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.347775    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:11.347853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.347828    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:11.348224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:11.348208    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:19:12.289635 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.289600    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq" podUID="91eeb679-7ad2-48a4-b977-f86ee4f4d788" containerName="payload-processing" containerID="cri-o://d7d8b6a647852cb2e49f5689d3b7ea2d6bf6da5cdfa3bcbabd903a3583321742" gracePeriod=25
Nov 03 12:19:12.439464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.439424    4546 generic.go:356] "Generic (PLEG): container finished" podID="91eeb679-7ad2-48a4-b977-f86ee4f4d788" containerID="d7d8b6a647852cb2e49f5689d3b7ea2d6bf6da5cdfa3bcbabd903a3583321742" exitCode=0
Nov 03 12:19:12.440076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.439514    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq" event={"ID":"91eeb679-7ad2-48a4-b977-f86ee4f4d788","Type":"ContainerDied","Data":"d7d8b6a647852cb2e49f5689d3b7ea2d6bf6da5cdfa3bcbabd903a3583321742"}
Nov 03 12:19:12.584391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.584360    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:19:12.649420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.649390    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume\") pod \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") "
Nov 03 12:19:12.649571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.649517    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb\" (UniqueName: \"kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb\") pod \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\" (UID: \"91eeb679-7ad2-48a4-b977-f86ee4f4d788\") "
Nov 03 12:19:12.649818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.649798    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume" pod "91eeb679-7ad2-48a4-b977-f86ee4f4d788" (UID: "91eeb679-7ad2-48a4-b977-f86ee4f4d788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:12.650118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.650100    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91eeb679-7ad2-48a4-b977-f86ee4f4d788-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:12.651705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.651681    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb" pod "91eeb679-7ad2-48a4-b977-f86ee4f4d788" (UID: "91eeb679-7ad2-48a4-b977-f86ee4f4d788"). InnerVolumeSpecName "kube-api-access-58ndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:12.751575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:12.751542    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ndb\" (UniqueName: \"kubernetes.io/projected/91eeb679-7ad2-48a4-b977-f86ee4f4d788-kube-api-access-58ndb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:13.428610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.428575    4546 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:13.429077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.428632    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:13.447703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.447676    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"
Nov 03 12:19:13.447862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.447706    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq" event={"ID":"91eeb679-7ad2-48a4-b977-f86ee4f4d788","Type":"ContainerDied","Data":"26ba933d8756b9dff1fcb8719aaebe68fe49251d0caa99f47ca2009027c5b2ce"}
Nov 03 12:19:13.447862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.447788    4546 scope.go:122] "RemoveContainer" containerID="d7d8b6a647852cb2e49f5689d3b7ea2d6bf6da5cdfa3bcbabd903a3583321742"
Nov 03 12:19:13.615782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.614892    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"]
Nov 03 12:19:13.636546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:13.636507    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-7d6d9b658f-qx4nq"]
Nov 03 12:19:14.574698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.574672    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91eeb679-7ad2-48a4-b977-f86ee4f4d788" path="/var/lib/kubelet/pods/91eeb679-7ad2-48a4-b977-f86ee4f4d788/volumes"
Nov 03 12:19:14.815148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.815119    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:19:14.859662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.859634    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:19:14.877844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.877794    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.877844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.877845    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.877884    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.878130    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.878157    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.878201    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.878233    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.878289    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.878377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.878333    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs\") pod \"b70d5924-b47a-43dd-b583-0a2e381350ee\" (UID: \"b70d5924-b47a-43dd-b583-0a2e381350ee\") "
Nov 03 12:19:14.879853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.879821    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.879976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.879957    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.880049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.880029    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.880162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.880136    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.880447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.880430    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:14.883879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.883856    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "kube-api-access-nf4r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:14.884274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.884252    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:14.884580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.884551    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.884659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.884583    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo" pod "b70d5924-b47a-43dd-b583-0a2e381350ee" (UID: "b70d5924-b47a-43dd-b583-0a2e381350ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:14.925800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.925778    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:19:14.928493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.928474    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:19:14.979471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979445    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979488    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979537    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979574    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.979614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979600    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.979874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979627    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979689    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979744    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979812    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.979874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979810    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.979874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979860    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979894    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.979965    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980005    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980047    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980140    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980166    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.980459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980225    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:14.980459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980234    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr\") pod \"611e4d34-68ad-454c-8088-5b42b559d36e\" (UID: \"611e4d34-68ad-454c-8088-5b42b559d36e\") "
Nov 03 12:19:14.980459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980357    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.980459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.980391    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket\") pod \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\" (UID: \"381ed6a4-0fd7-47e4-95a8-9a6c119b3743\") "
Nov 03 12:19:14.981440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981415    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981445    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70d5924-b47a-43dd-b583-0a2e381350ee-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981462    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981476    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981492    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981507    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981521    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981534    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981549    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4r4\" (UniqueName: \"kubernetes.io/projected/b70d5924-b47a-43dd-b583-0a2e381350ee-kube-api-access-nf4r4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981564    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70d5924-b47a-43dd-b583-0a2e381350ee-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981578    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981591    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611e4d34-68ad-454c-8088-5b42b559d36e-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:14.981954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.981887    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.982258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982200    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:14.982493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982468    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.982580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982529    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:14.982680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982662    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.983156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982812    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.983156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982950    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:14.983156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982957    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:14.983156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.982963    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.983156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.983132    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.983428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.983318    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "kube-api-access-2sq5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:14.984625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.984601    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "kube-api-access-fcnnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:14.985436 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.985413    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:14.985535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.985491    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo" pod "381ed6a4-0fd7-47e4-95a8-9a6c119b3743" (UID: "381ed6a4-0fd7-47e4-95a8-9a6c119b3743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:14.986843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:14.986824    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy" pod "611e4d34-68ad-454c-8088-5b42b559d36e" (UID: "611e4d34-68ad-454c-8088-5b42b559d36e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.082763    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.082831    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.082868    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083000    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083072    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083120    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083200    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083253    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083282    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket\") pod \"9b43c591-0ba5-419f-8bf0-55c164c532e2\" (UID: \"9b43c591-0ba5-419f-8bf0-55c164c532e2\") "
Nov 03 12:19:15.083586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083467    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.083672    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084075    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084071    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084121    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084137    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084151    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084163    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084193    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084207    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611e4d34-68ad-454c-8088-5b42b559d36e-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084219    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcnnr\" (UniqueName: \"kubernetes.io/projected/611e4d34-68ad-454c-8088-5b42b559d36e-kube-api-access-fcnnr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084232    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b43c591-0ba5-419f-8bf0-55c164c532e2-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084244    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084276    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084288    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sq5l\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-kube-api-access-2sq5l\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084299    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084312    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611e4d34-68ad-454c-8088-5b42b559d36e-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084325    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084355    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.084616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084367    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381ed6a4-0fd7-47e4-95a8-9a6c119b3743-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.086381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084643    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:15.086381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.084899    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:15.088230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.088190    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:15.088623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.088590    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:15.088801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.088777    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:15.090390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.090365    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85" pod "9b43c591-0ba5-419f-8bf0-55c164c532e2" (UID: "9b43c591-0ba5-419f-8bf0-55c164c532e2"). InnerVolumeSpecName "kube-api-access-v8d85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:15.185649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185623    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8d85\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-kube-api-access-v8d85\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.185649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185648    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.185848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185659    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.185848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185668    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.185848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185682    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.185848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185694    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.185848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.185701    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b43c591-0ba5-419f-8bf0-55c164c532e2-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:15.463780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.463661    4546 generic.go:356] "Generic (PLEG): container finished" podID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerID="2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd" exitCode=0
Nov 03 12:19:15.463780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.463693    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" event={"ID":"b70d5924-b47a-43dd-b583-0a2e381350ee","Type":"ContainerDied","Data":"2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd"}
Nov 03 12:19:15.464012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.463717    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"
Nov 03 12:19:15.464012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.463811    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj" event={"ID":"b70d5924-b47a-43dd-b583-0a2e381350ee","Type":"ContainerDied","Data":"34bb8cb39aa69e17bcdf65b5b31015b8ffa0aed6944d76e50c4ef0199225d853"}
Nov 03 12:19:15.464012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.463840    4546 scope.go:122] "RemoveContainer" containerID="2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd"
Nov 03 12:19:15.465764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.465724    4546 generic.go:356] "Generic (PLEG): container finished" podID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerID="aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614" exitCode=0
Nov 03 12:19:15.465888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.465784    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" event={"ID":"9b43c591-0ba5-419f-8bf0-55c164c532e2","Type":"ContainerDied","Data":"aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614"}
Nov 03 12:19:15.465888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.465812    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"
Nov 03 12:19:15.465888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.465832    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn" event={"ID":"9b43c591-0ba5-419f-8bf0-55c164c532e2","Type":"ContainerDied","Data":"14bdc6cddd09f1fe015c61e7fcd1dc4c13f7eea2c0abf7fabe93bfcb800a227e"}
Nov 03 12:19:15.467369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.467345    4546 generic.go:356] "Generic (PLEG): container finished" podID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerID="c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a" exitCode=0
Nov 03 12:19:15.467475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.467391    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"
Nov 03 12:19:15.467475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.467434    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" event={"ID":"381ed6a4-0fd7-47e4-95a8-9a6c119b3743","Type":"ContainerDied","Data":"c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a"}
Nov 03 12:19:15.467611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.467484    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz" event={"ID":"381ed6a4-0fd7-47e4-95a8-9a6c119b3743","Type":"ContainerDied","Data":"efdd979a977acee7548664d8454e433c375e40e69aac6668b83cbef9a7936b53"}
Nov 03 12:19:15.469062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.469041    4546 generic.go:356] "Generic (PLEG): container finished" podID="611e4d34-68ad-454c-8088-5b42b559d36e" containerID="3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304" exitCode=0
Nov 03 12:19:15.469158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.469075    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" event={"ID":"611e4d34-68ad-454c-8088-5b42b559d36e","Type":"ContainerDied","Data":"3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304"}
Nov 03 12:19:15.469158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.469097    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"
Nov 03 12:19:15.469158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.469107    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv" event={"ID":"611e4d34-68ad-454c-8088-5b42b559d36e","Type":"ContainerDied","Data":"c506d36b7f75e10d15bfd5f6183aafa66b188babdcbac566763f35982228b062"}
Nov 03 12:19:15.476671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.476653    4546 scope.go:122] "RemoveContainer" containerID="2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd"
Nov 03 12:19:15.476987 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:15.476966    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd\": container with ID starting with 2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd not found: ID does not exist" containerID="2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd"
Nov 03 12:19:15.477081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.476997    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd"} err="failed to get container status \"2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd\": rpc error: code = NotFound desc = could not find container \"2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd\": container with ID starting with 2ecad457cf2b8f2659a05cb57803afc05c777015b4e4221703b8d557847fb7dd not found: ID does not exist"
Nov 03 12:19:15.477081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.477017    4546 scope.go:122] "RemoveContainer" containerID="aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614"
Nov 03 12:19:15.516890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.516872    4546 scope.go:122] "RemoveContainer" containerID="37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"
Nov 03 12:19:15.525905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.525881    4546 scope.go:122] "RemoveContainer" containerID="aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614"
Nov 03 12:19:15.526152 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:15.526135    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614\": container with ID starting with aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614 not found: ID does not exist" containerID="aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614"
Nov 03 12:19:15.526224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.526159    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614"} err="failed to get container status \"aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614\": rpc error: code = NotFound desc = could not find container \"aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614\": container with ID starting with aa4572e055011d844486bac999f07ed3e736553f7e2904364617c1f108353614 not found: ID does not exist"
Nov 03 12:19:15.526224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.526175    4546 scope.go:122] "RemoveContainer" containerID="37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"
Nov 03 12:19:15.526416 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:15.526401    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae\": container with ID starting with 37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae not found: ID does not exist" containerID="37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"
Nov 03 12:19:15.526462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.526423    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae"} err="failed to get container status \"37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae\": rpc error: code = NotFound desc = could not find container \"37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae\": container with ID starting with 37eab851117aca7c049e9cc067cf01a0ebd7a6c6dc1b86e9605d7bc47e25edae not found: ID does not exist"
Nov 03 12:19:15.526462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.526438    4546 scope.go:122] "RemoveContainer" containerID="c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a"
Nov 03 12:19:15.535162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.535146    4546 scope.go:122] "RemoveContainer" containerID="c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a"
Nov 03 12:19:15.535403 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:15.535385    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a\": container with ID starting with c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a not found: ID does not exist" containerID="c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a"
Nov 03 12:19:15.535475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.535406    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a"} err="failed to get container status \"c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a\": rpc error: code = NotFound desc = could not find container \"c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a\": container with ID starting with c8956d61cf471c703720768ff13728f62797424bd1c773038741da8ba8edf79a not found: ID does not exist"
Nov 03 12:19:15.535475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.535420    4546 scope.go:122] "RemoveContainer" containerID="3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304"
Nov 03 12:19:15.544036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.544023    4546 scope.go:122] "RemoveContainer" containerID="3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304"
Nov 03 12:19:15.544299 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:15.544282    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304\": container with ID starting with 3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304 not found: ID does not exist" containerID="3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304"
Nov 03 12:19:15.544341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.544308    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304"} err="failed to get container status \"3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304\": rpc error: code = NotFound desc = could not find container \"3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304\": container with ID starting with 3068fc0a65966f464f0cc8ba70c82817ad52c54a7e0337e71880c9cacd427304 not found: ID does not exist"
Nov 03 12:19:15.617684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.617662    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"]
Nov 03 12:19:15.621064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.621046    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-wvwjz"]
Nov 03 12:19:15.722293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.722210    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"]
Nov 03 12:19:15.726365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.726340    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-58lzj"]
Nov 03 12:19:15.828137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.828096    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"]
Nov 03 12:19:15.831569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.831547    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-5g7nn"]
Nov 03 12:19:15.981173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.981098    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"]
Nov 03 12:19:15.984555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:15.984531    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vz8rv"]
Nov 03 12:19:16.573601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:16.573573    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" path="/var/lib/kubelet/pods/381ed6a4-0fd7-47e4-95a8-9a6c119b3743/volumes"
Nov 03 12:19:16.574078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:16.574061    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" path="/var/lib/kubelet/pods/611e4d34-68ad-454c-8088-5b42b559d36e/volumes"
Nov 03 12:19:16.574478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:16.574466    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" path="/var/lib/kubelet/pods/9b43c591-0ba5-419f-8bf0-55c164c532e2/volumes"
Nov 03 12:19:16.575193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:16.575179    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" path="/var/lib/kubelet/pods/b70d5924-b47a-43dd-b583-0a2e381350ee/volumes"
Nov 03 12:19:17.391106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391068    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"]
Nov 03 12:19:17.391815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391794    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy"
Nov 03 12:19:17.391815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391813    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391827    4546 cpu_manager.go:414] "containerMap: removing container" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391833    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391849    4546 cpu_manager.go:414] "containerMap: removing container" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391854    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391873    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391877    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391884    4546 cpu_manager.go:414] "containerMap: removing container" podUID="91eeb679-7ad2-48a4-b977-f86ee4f4d788" containerName="payload-processing"
Nov 03 12:19:17.391965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391888    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91eeb679-7ad2-48a4-b977-f86ee4f4d788" containerName="payload-processing"
Nov 03 12:19:17.392230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391982    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70d5924-b47a-43dd-b583-0a2e381350ee" containerName="istio-proxy"
Nov 03 12:19:17.392230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391991    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="91eeb679-7ad2-48a4-b977-f86ee4f4d788" containerName="payload-processing"
Nov 03 12:19:17.392230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.391998    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy"
Nov 03 12:19:17.392230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.392006    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy"
Nov 03 12:19:17.392230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.392013    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="381ed6a4-0fd7-47e4-95a8-9a6c119b3743" containerName="istio-proxy"
Nov 03 12:19:17.392230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.392020    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="611e4d34-68ad-454c-8088-5b42b559d36e" containerName="istio-proxy"
Nov 03 12:19:17.395455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.395436    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.449733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.449702    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"]
Nov 03 12:19:17.501774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.493197    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45b1b7-dockercfg-kptzn\""
Nov 03 12:19:17.501774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.493263    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45b1b7\""
Nov 03 12:19:17.515557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.515526    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpcs\" (UniqueName: \"kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.515735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.515678    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.561385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.561355    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"]
Nov 03 12:19:17.562220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.562197    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy"
Nov 03 12:19:17.562220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.562218    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b43c591-0ba5-419f-8bf0-55c164c532e2" containerName="istio-proxy"
Nov 03 12:19:17.566363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.566343    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.619978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.619942    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.620322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.620302    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.620667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.620637    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwnp\" (UniqueName: \"kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp\") pod \"payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.620894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.620843    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpcs\" (UniqueName: \"kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.623220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.623197    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.671988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.671832    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"]
Nov 03 12:19:17.681082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.680379    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpcs\" (UniqueName: \"kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.713197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.713164    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:17.722263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.722234    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwnp\" (UniqueName: \"kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp\") pod \"payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.722404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.722337    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.723069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.723047    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.781138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.778217    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwnp\" (UniqueName: \"kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp\") pod \"payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:17.881702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:17.881668    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:18.094949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.094888    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"]
Nov 03 12:19:18.095991 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:18.095958    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6120c16_264d_4a54_b4a3_7ba38464dbb7.slice/crio-38c113649492ae35124a8fd5bb662d71a47e0057c8249ccd0cb8f076a1815743 WatchSource:0}: Error finding container 38c113649492ae35124a8fd5bb662d71a47e0057c8249ccd0cb8f076a1815743: Status 404 returned error can't find the container with id 38c113649492ae35124a8fd5bb662d71a47e0057c8249ccd0cb8f076a1815743
Nov 03 12:19:18.250737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.250712    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"]
Nov 03 12:19:18.251593 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:18.251571    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c640c9_fcb6_4c15_bbbd_11787a1fa5f0.slice/crio-ae43c2c42b81a515e55d1e4d1dc7169591e406d098cb78a56a904f241e856e80 WatchSource:0}: Error finding container ae43c2c42b81a515e55d1e4d1dc7169591e406d098cb78a56a904f241e856e80: Status 404 returned error can't find the container with id ae43c2c42b81a515e55d1e4d1dc7169591e406d098cb78a56a904f241e856e80
Nov 03 12:19:18.511951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.511865    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc" event={"ID":"e6120c16-264d-4a54-b4a3-7ba38464dbb7","Type":"ContainerStarted","Data":"b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a"}
Nov 03 12:19:18.511951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.511924    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc" event={"ID":"e6120c16-264d-4a54-b4a3-7ba38464dbb7","Type":"ContainerStarted","Data":"38c113649492ae35124a8fd5bb662d71a47e0057c8249ccd0cb8f076a1815743"}
Nov 03 12:19:18.512472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.512101    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:18.513611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.513586    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8" event={"ID":"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0","Type":"ContainerStarted","Data":"a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5"}
Nov 03 12:19:18.513695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.513636    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:18.513695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.513657    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8" event={"ID":"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0","Type":"ContainerStarted","Data":"ae43c2c42b81a515e55d1e4d1dc7169591e406d098cb78a56a904f241e856e80"}
Nov 03 12:19:18.619763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.619689    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc" podStartSLOduration=1.61967618 podStartE2EDuration="1.61967618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:18.61692403 +0000 UTC m=+1700.188507959" watchObservedRunningTime="2026-11-03 12:19:18.61967618 +0000 UTC m=+1700.191260091"
Nov 03 12:19:18.721892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:18.721848    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8" podStartSLOduration=1.721835601 podStartE2EDuration="1.721835601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:18.721518923 +0000 UTC m=+1700.293102855" watchObservedRunningTime="2026-11-03 12:19:18.721835601 +0000 UTC m=+1700.293419511"
Nov 03 12:19:24.912653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:24.912617    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"]
Nov 03 12:19:25.010563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:25.010504    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"]
Nov 03 12:19:26.818616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.818580    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"]
Nov 03 12:19:26.819092 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.818861    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy" containerID="cri-o://a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c" gracePeriod=30
Nov 03 12:19:26.864530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.864491    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"]
Nov 03 12:19:26.864828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.864742    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" containerID="cri-o://602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80" gracePeriod=30
Nov 03 12:19:26.867902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.867879    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"]
Nov 03 12:19:26.868082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.868060    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" containerID="cri-o://291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83" gracePeriod=30
Nov 03 12:19:26.909957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.909927    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"]
Nov 03 12:19:26.910121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:26.910102    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" containerID="cri-o://57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c" gracePeriod=30
Nov 03 12:19:27.221375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.221346    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"]
Nov 03 12:19:27.226872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.226851    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.237855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.237818    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"]
Nov 03 12:19:27.242028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.242007    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.280870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.280846    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"]
Nov 03 12:19:27.304020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.303986    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"]
Nov 03 12:19:27.343880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.343845    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.343894    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.343987    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5nf\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344045    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344139    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344224    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344300    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344358    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344396    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344484    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzj7\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344527    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344555    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344690    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.344817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344803    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344841    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344947    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.344995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.344988    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.345042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.345006    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.446676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446628    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.446676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446681    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.446925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446739    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.446925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446825    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.446925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446852    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446952    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.446981    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.447081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447008    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5nf\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447051    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447080    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447130    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447176    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447184    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447198    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447213    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.447277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447252    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447290    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzj7\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.447657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447333    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447358    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447428    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447535    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.447973 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.447947    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.448199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.448153    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.448199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.448165    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.448380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.448254    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.448380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.448281    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.448380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.448331    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.448380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.448338    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.449432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.449412    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.449795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.449778    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.449855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.449839    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.450149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.450125    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.499803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.499718    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.499917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.499812    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.500128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.500082    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzj7\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.500524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.500481    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5nf\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.542242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.542212    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:27.554562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.554538    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:27.966459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.966435    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"]
Nov 03 12:19:27.968130 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:27.968098    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9760034f_aaab_4f63_a7f6_c5740e1ff5b0.slice/crio-8fbc52395ebb3905b4d77388625a63385b6ee8e9ed0b17f3f0adf6c2062b5050 WatchSource:0}: Error finding container 8fbc52395ebb3905b4d77388625a63385b6ee8e9ed0b17f3f0adf6c2062b5050: Status 404 returned error can't find the container with id 8fbc52395ebb3905b4d77388625a63385b6ee8e9ed0b17f3f0adf6c2062b5050
Nov 03 12:19:27.968990 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:27.968969    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431cccbb_c8e4_4b47_9e87_97819e146a4a.slice/crio-0bd38c875de6665762ed0a14e4bdf0b2e236142d8f6ee5a4d4d2190b6d0df4ee WatchSource:0}: Error finding container 0bd38c875de6665762ed0a14e4bdf0b2e236142d8f6ee5a4d4d2190b6d0df4ee: Status 404 returned error can't find the container with id 0bd38c875de6665762ed0a14e4bdf0b2e236142d8f6ee5a4d4d2190b6d0df4ee
Nov 03 12:19:27.970473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.970452    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"]
Nov 03 12:19:27.970698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.970666    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:27.970794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.970732    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:27.970794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.970786    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:27.971255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.971163    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:27.971255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.971231    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:27.971362 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:27.971271    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:28.006004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.005977    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:28.006109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.006031    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:28.677946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.661634    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" event={"ID":"9760034f-aaab-4f63-a7f6-c5740e1ff5b0","Type":"ContainerStarted","Data":"316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38"}
Nov 03 12:19:28.678177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.678141    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" event={"ID":"9760034f-aaab-4f63-a7f6-c5740e1ff5b0","Type":"ContainerStarted","Data":"8fbc52395ebb3905b4d77388625a63385b6ee8e9ed0b17f3f0adf6c2062b5050"}
Nov 03 12:19:28.678256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.678188    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" event={"ID":"431cccbb-c8e4-4b47-9e87-97819e146a4a","Type":"ContainerStarted","Data":"2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79"}
Nov 03 12:19:28.678256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.678235    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" event={"ID":"431cccbb-c8e4-4b47-9e87-97819e146a4a","Type":"ContainerStarted","Data":"0bd38c875de6665762ed0a14e4bdf0b2e236142d8f6ee5a4d4d2190b6d0df4ee"}
Nov 03 12:19:28.750147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:28.750084    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podStartSLOduration=1.750068108 podStartE2EDuration="1.750068108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:28.749294222 +0000 UTC m=+1710.320878178" watchObservedRunningTime="2026-11-03 12:19:28.750068108 +0000 UTC m=+1710.321652019"
Nov 03 12:19:29.546916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.546885    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:29.547672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.547648    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:19:29.547856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.547830    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 12:19:29.556782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.554930    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:19:29.557881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.557844    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:29.558975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.558109    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:19:29.558975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.558856    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 12:19:29.558975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.558900    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 12:19:29.665569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.665000    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podStartSLOduration=2.664982254 podStartE2EDuration="2.664982254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:28.861103213 +0000 UTC m=+1710.432687170" watchObservedRunningTime="2026-11-03 12:19:29.664982254 +0000 UTC m=+1711.236566169"
Nov 03 12:19:29.920836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:29.914890    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh" podUID="2a533ac7-f535-4725-bb9d-c7f3a76f44fd" containerName="payload-processing" containerID="cri-o://b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a" gracePeriod=25
Nov 03 12:19:30.021778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.020160    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9" podUID="93f10df6-086e-475b-bbd5-624329ba890e" containerName="payload-pre-processing" containerID="cri-o://ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6" gracePeriod=25
Nov 03 12:19:30.342538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.342201    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:30.342538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.342260    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:30.466387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.466009    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:19:30.495703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.495680    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr\" (UniqueName: \"kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr\") pod \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") "
Nov 03 12:19:30.495994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.495975    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume\") pod \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\" (UID: \"2a533ac7-f535-4725-bb9d-c7f3a76f44fd\") "
Nov 03 12:19:30.496460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.496435    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume" pod "2a533ac7-f535-4725-bb9d-c7f3a76f44fd" (UID: "2a533ac7-f535-4725-bb9d-c7f3a76f44fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:30.497994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.497968    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr" pod "2a533ac7-f535-4725-bb9d-c7f3a76f44fd" (UID: "2a533ac7-f535-4725-bb9d-c7f3a76f44fd"). InnerVolumeSpecName "kube-api-access-2fvmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:30.543581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.543139    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:19:30.543581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.543212    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 12:19:30.559766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.557322    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 12:19:30.559766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.557372    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 12:19:30.562643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.562351    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:19:30.562643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.562482    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:19:30.567540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.567511    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:19:30.598655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.598626    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr\" (UniqueName: \"kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr\") pod \"93f10df6-086e-475b-bbd5-624329ba890e\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") "
Nov 03 12:19:30.598945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.598920    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume\") pod \"93f10df6-086e-475b-bbd5-624329ba890e\" (UID: \"93f10df6-086e-475b-bbd5-624329ba890e\") "
Nov 03 12:19:30.599474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.599444    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume" pod "93f10df6-086e-475b-bbd5-624329ba890e" (UID: "93f10df6-086e-475b-bbd5-624329ba890e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:30.599555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.599515    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:30.599555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.599536    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvmr\" (UniqueName: \"kubernetes.io/projected/2a533ac7-f535-4725-bb9d-c7f3a76f44fd-kube-api-access-2fvmr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:30.601277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.601213    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr" pod "93f10df6-086e-475b-bbd5-624329ba890e" (UID: "93f10df6-086e-475b-bbd5-624329ba890e"). InnerVolumeSpecName "kube-api-access-8wqvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:30.692061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.692030    4546 generic.go:356] "Generic (PLEG): container finished" podID="93f10df6-086e-475b-bbd5-624329ba890e" containerID="ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6" exitCode=0
Nov 03 12:19:30.692291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.692096    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9" event={"ID":"93f10df6-086e-475b-bbd5-624329ba890e","Type":"ContainerDied","Data":"ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6"}
Nov 03 12:19:30.692291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.692187    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9" event={"ID":"93f10df6-086e-475b-bbd5-624329ba890e","Type":"ContainerDied","Data":"71ec4943fced834f7360d19fcc4862b859e9bba187aa2e9f6934e252b998a034"}
Nov 03 12:19:30.692291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.692212    4546 scope.go:122] "RemoveContainer" containerID="ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6"
Nov 03 12:19:30.692609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.692591    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"
Nov 03 12:19:30.693938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.693896    4546 generic.go:356] "Generic (PLEG): container finished" podID="2a533ac7-f535-4725-bb9d-c7f3a76f44fd" containerID="b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a" exitCode=0
Nov 03 12:19:30.694015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.693938    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"
Nov 03 12:19:30.694015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.693944    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh" event={"ID":"2a533ac7-f535-4725-bb9d-c7f3a76f44fd","Type":"ContainerDied","Data":"b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a"}
Nov 03 12:19:30.694015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.693994    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh" event={"ID":"2a533ac7-f535-4725-bb9d-c7f3a76f44fd","Type":"ContainerDied","Data":"d43f064692a9a44af80420ff37cac8fa1b651f9aad814293fd6aa18dec7623e0"}
Nov 03 12:19:30.700282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.700259    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f10df6-086e-475b-bbd5-624329ba890e-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:30.700358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.700289    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqvr\" (UniqueName: \"kubernetes.io/projected/93f10df6-086e-475b-bbd5-624329ba890e-kube-api-access-8wqvr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:30.705211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.705193    4546 scope.go:122] "RemoveContainer" containerID="ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6"
Nov 03 12:19:30.705471 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:30.705451    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6\": container with ID starting with ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6 not found: ID does not exist" containerID="ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6"
Nov 03 12:19:30.705542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.705483    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6"} err="failed to get container status \"ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6\": rpc error: code = NotFound desc = could not find container \"ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6\": container with ID starting with ede3f1f9f245cbf22f5d8fce7a0d0b105c93bcf951917e25a26ac2d9a99ae6c6 not found: ID does not exist"
Nov 03 12:19:30.705542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.705507    4546 scope.go:122] "RemoveContainer" containerID="b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a"
Nov 03 12:19:30.771597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.771571    4546 scope.go:122] "RemoveContainer" containerID="b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a"
Nov 03 12:19:30.771952 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:30.771933    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a\": container with ID starting with b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a not found: ID does not exist" containerID="b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a"
Nov 03 12:19:30.772008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.771960    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a"} err="failed to get container status \"b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a\": rpc error: code = NotFound desc = could not find container \"b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a\": container with ID starting with b2ed3d41a023e9a162b72fec280981fbd7187c0c1837355ee39ade30d91f7c5a not found: ID does not exist"
Nov 03 12:19:30.872397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.872302    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"]
Nov 03 12:19:30.877492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.877457    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6c034-cb7d9d575-wdpnh"]
Nov 03 12:19:30.976725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.976688    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"]
Nov 03 12:19:30.980111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:30.980087    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6c034-86cf4d579-tnnh9"]
Nov 03 12:19:31.546206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:31.546170    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:19:31.546499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:31.546469    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 12:19:31.557160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:31.557113    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 12:19:31.557386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:31.557361    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 12:19:32.372790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.372287    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.478736    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.478854    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.478931    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.478959    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.479050    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.479159    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.479244    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.479288    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.479594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.479365    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs\") pod \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\" (UID: \"7ed6b079-a27d-4a6d-a49d-52cefe98ee71\") "
Nov 03 12:19:32.480673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.480548    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.480799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.480778    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.480888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.480866    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:32.481157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.481131    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.483176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.483001    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.485383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.485354    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:32.486691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.486668    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.487536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.487501    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.489912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.488601    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2" pod "7ed6b079-a27d-4a6d-a49d-52cefe98ee71" (UID: "7ed6b079-a27d-4a6d-a49d-52cefe98ee71"). InnerVolumeSpecName "kube-api-access-sd6b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.493821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.493738    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:19:32.570340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.570308    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:19:32.570484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.570368    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 12:19:32.570799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.570776    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 12:19:32.570905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.570816    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 12:19:32.581661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.581632    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a533ac7-f535-4725-bb9d-c7f3a76f44fd" path="/var/lib/kubelet/pods/2a533ac7-f535-4725-bb9d-c7f3a76f44fd/volumes"
Nov 03 12:19:32.582270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.582248    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f10df6-086e-475b-bbd5-624329ba890e" path="/var/lib/kubelet/pods/93f10df6-086e-475b-bbd5-624329ba890e/volumes"
Nov 03 12:19:32.583739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.583717    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584155    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584267    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584363    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6b2\" (UniqueName: \"kubernetes.io/projected/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-kube-api-access-sd6b2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584455    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584541    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584633    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584740    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.584881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.584868    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed6b079-a27d-4a6d-a49d-52cefe98ee71-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.595076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.591789    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:19:32.595076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.593730    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:19:32.686334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.686298    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.686525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.686348    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.686636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.686593    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.686700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.686665    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.687094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.686741    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.687151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687128    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.687195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687163    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.687227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687201    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.687487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687466    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.687713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687691    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.687801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687769    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.687865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687815    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.687865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687860    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.687975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687909    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.687975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687961    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.688075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.687996    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.688273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688251    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.688524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688027    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.688604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688561    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.688639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688608    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket\") pod \"511d427a-a19d-411f-9c7c-2fd4674f2c73\" (UID: \"511d427a-a19d-411f-9c7c-2fd4674f2c73\") "
Nov 03 12:19:32.688670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688647    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.688721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688706    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.688788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688775    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.688859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.688809    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.689824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.689309    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:32.689824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.689332    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:32.689824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.689568    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.689824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.689601    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.689824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.689792    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.690325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.690294    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:32.690738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.690714    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.691157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.691129    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.691412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.691247    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:32.691412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.691335    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.691678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.691624    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.691951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.691368    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.692045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.692030    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.692326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.692301    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.692540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.692510    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.692696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.692678    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.693206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.692854    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.693422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.692957    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "kube-api-access-wkzgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.693422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.693173    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:32.693422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.693162    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "kube-api-access-ncc78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.693952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.693893    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.694232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.694101    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.694232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.694209    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "kube-api-access-xrpxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.694817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.693364    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data\") pod \"57fec5de-3713-448e-a3aa-18c43d550178\" (UID: \"57fec5de-3713-448e-a3aa-18c43d550178\") "
Nov 03 12:19:32.694817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.694481    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token\") pod \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\" (UID: \"dc0e3ad4-23fd-4535-bef5-a2a475c52d95\") "
Nov 03 12:19:32.695329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.695044    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696296    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696473    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696479    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696486    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696518    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696536    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696551    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncc78\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-kube-api-access-ncc78\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696566    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696578    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696590    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696603    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696616    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzgx\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-kube-api-access-wkzgx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696630    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/511d427a-a19d-411f-9c7c-2fd4674f2c73-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696642    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696655    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696668    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fec5de-3713-448e-a3aa-18c43d550178-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696681    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696695    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696707    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696719    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696731    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpxx\" (UniqueName: \"kubernetes.io/projected/57fec5de-3713-448e-a3aa-18c43d550178-kube-api-access-xrpxx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696764    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696779    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.696792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.696791    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fec5de-3713-448e-a3aa-18c43d550178-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.697669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.697070    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token" pod "dc0e3ad4-23fd-4535-bef5-a2a475c52d95" (UID: "dc0e3ad4-23fd-4535-bef5-a2a475c52d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:32.700512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.700488    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy" pod "511d427a-a19d-411f-9c7c-2fd4674f2c73" (UID: "511d427a-a19d-411f-9c7c-2fd4674f2c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:19:32.701832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.701808    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo" pod "57fec5de-3713-448e-a3aa-18c43d550178" (UID: "57fec5de-3713-448e-a3aa-18c43d550178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:19:32.764773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.764721    4546 generic.go:356] "Generic (PLEG): container finished" podID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerID="602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80" exitCode=0
Nov 03 12:19:32.765036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.764797    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"
Nov 03 12:19:32.765036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.764846    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" event={"ID":"511d427a-a19d-411f-9c7c-2fd4674f2c73","Type":"ContainerDied","Data":"602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80"}
Nov 03 12:19:32.765036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.764910    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg" event={"ID":"511d427a-a19d-411f-9c7c-2fd4674f2c73","Type":"ContainerDied","Data":"771c75bbba3be0190ac957a2cf2d57d69158d3ad5fc79061ec6536c43bfd7992"}
Nov 03 12:19:32.765200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.765114    4546 scope.go:122] "RemoveContainer" containerID="602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80"
Nov 03 12:19:32.766874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.766775    4546 generic.go:356] "Generic (PLEG): container finished" podID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerID="a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c" exitCode=0
Nov 03 12:19:32.766962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.766899    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" event={"ID":"7ed6b079-a27d-4a6d-a49d-52cefe98ee71","Type":"ContainerDied","Data":"a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c"}
Nov 03 12:19:32.766962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.766921    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"
Nov 03 12:19:32.766962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.766940    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59" event={"ID":"7ed6b079-a27d-4a6d-a49d-52cefe98ee71","Type":"ContainerDied","Data":"c107b52a83bf3c0423509098910752e4980729adb5cdea7fe88b98c30d19890e"}
Nov 03 12:19:32.768867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.768811    4546 generic.go:356] "Generic (PLEG): container finished" podID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerID="291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83" exitCode=0
Nov 03 12:19:32.768867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.768853    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"
Nov 03 12:19:32.769109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.769081    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" event={"ID":"dc0e3ad4-23fd-4535-bef5-a2a475c52d95","Type":"ContainerDied","Data":"291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83"}
Nov 03 12:19:32.769173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.769140    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c" event={"ID":"dc0e3ad4-23fd-4535-bef5-a2a475c52d95","Type":"ContainerDied","Data":"919789aa1385c2b56e088ddee01263710128473afc4e195d8c3705fe1666d10a"}
Nov 03 12:19:32.771140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.771118    4546 generic.go:356] "Generic (PLEG): container finished" podID="57fec5de-3713-448e-a3aa-18c43d550178" containerID="57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c" exitCode=0
Nov 03 12:19:32.771237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.771147    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" event={"ID":"57fec5de-3713-448e-a3aa-18c43d550178","Type":"ContainerDied","Data":"57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c"}
Nov 03 12:19:32.771237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.771157    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"
Nov 03 12:19:32.771237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.771178    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk" event={"ID":"57fec5de-3713-448e-a3aa-18c43d550178","Type":"ContainerDied","Data":"818c747f59bceaa53778b7067adc4656afe33c440ab137be8ffaac3cd2086c21"}
Nov 03 12:19:32.777637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.777620    4546 scope.go:122] "RemoveContainer" containerID="f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"
Nov 03 12:19:32.791176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.791160    4546 scope.go:122] "RemoveContainer" containerID="602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80"
Nov 03 12:19:32.792011 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:32.791984    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80\": container with ID starting with 602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80 not found: ID does not exist" containerID="602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80"
Nov 03 12:19:32.792086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.792022    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80"} err="failed to get container status \"602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80\": rpc error: code = NotFound desc = could not find container \"602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80\": container with ID starting with 602c1160f7d32ac8ebc734a159b60350e4fea541ef69c53a9211efddee75fd80 not found: ID does not exist"
Nov 03 12:19:32.792086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.792044    4546 scope.go:122] "RemoveContainer" containerID="f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"
Nov 03 12:19:32.792297 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:32.792281    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec\": container with ID starting with f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec not found: ID does not exist" containerID="f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"
Nov 03 12:19:32.792340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.792305    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec"} err="failed to get container status \"f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec\": rpc error: code = NotFound desc = could not find container \"f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec\": container with ID starting with f6c09e43f6df5added117fb2af7c65e00cd297cf4d2dce56d583442872d182ec not found: ID does not exist"
Nov 03 12:19:32.792340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.792320    4546 scope.go:122] "RemoveContainer" containerID="a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c"
Nov 03 12:19:32.798638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.798610    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.798763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.798640    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0e3ad4-23fd-4535-bef5-a2a475c52d95-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.798763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.798654    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/511d427a-a19d-411f-9c7c-2fd4674f2c73-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.798763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.798668    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fec5de-3713-448e-a3aa-18c43d550178-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:32.802887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.802804    4546 scope.go:122] "RemoveContainer" containerID="a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c"
Nov 03 12:19:32.803367 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:32.803137    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c\": container with ID starting with a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c not found: ID does not exist" containerID="a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c"
Nov 03 12:19:32.803367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.803160    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c"} err="failed to get container status \"a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c\": rpc error: code = NotFound desc = could not find container \"a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c\": container with ID starting with a14c1652cb45538a1cb97ed73117e25ab0f7feb72eec1007b0a57f5a6134988c not found: ID does not exist"
Nov 03 12:19:32.803367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.803175    4546 scope.go:122] "RemoveContainer" containerID="291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83"
Nov 03 12:19:32.869201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.869174    4546 scope.go:122] "RemoveContainer" containerID="291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83"
Nov 03 12:19:32.869527 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:32.869506    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83\": container with ID starting with 291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83 not found: ID does not exist" containerID="291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83"
Nov 03 12:19:32.869584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.869534    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83"} err="failed to get container status \"291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83\": rpc error: code = NotFound desc = could not find container \"291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83\": container with ID starting with 291285a8c81c53a09f642b63b08b80ac9dd09933c0611ae587bb754dd0f04c83 not found: ID does not exist"
Nov 03 12:19:32.869584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.869554    4546 scope.go:122] "RemoveContainer" containerID="57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c"
Nov 03 12:19:32.890183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.890163    4546 scope.go:122] "RemoveContainer" containerID="57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c"
Nov 03 12:19:32.890562 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:32.890541    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c\": container with ID starting with 57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c not found: ID does not exist" containerID="57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c"
Nov 03 12:19:32.890643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:32.890568    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c"} err="failed to get container status \"57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c\": rpc error: code = NotFound desc = could not find container \"57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c\": container with ID starting with 57e863b85e8975b7cc87bb692444c868f326989770bfdb97561381785fa86c5c not found: ID does not exist"
Nov 03 12:19:33.042775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.041812    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"]
Nov 03 12:19:33.066775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.064342    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-r277c"]
Nov 03 12:19:33.137171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.137084    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"]
Nov 03 12:19:33.145009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.144972    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-ndl59"]
Nov 03 12:19:33.261846 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.261812    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"]
Nov 03 12:19:33.265160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.265124    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-bq4rk"]
Nov 03 12:19:33.363399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.363369    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"]
Nov 03 12:19:33.366178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.366153    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-vq9bg"]
Nov 03 12:19:33.545662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.545180    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 12:19:33.545662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.545257    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 12:19:33.556139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.556105    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 12:19:33.556293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:33.556176    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 12:19:34.547341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.547314    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:34.547799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.547779    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:34.548645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.548627    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:19:34.560649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.560626    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:34.561047 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.561023    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:34.562324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.562303    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:19:34.574368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.574334    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" path="/var/lib/kubelet/pods/511d427a-a19d-411f-9c7c-2fd4674f2c73/volumes"
Nov 03 12:19:34.574850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.574834    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fec5de-3713-448e-a3aa-18c43d550178" path="/var/lib/kubelet/pods/57fec5de-3713-448e-a3aa-18c43d550178/volumes"
Nov 03 12:19:34.575273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.575258    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" path="/var/lib/kubelet/pods/7ed6b079-a27d-4a6d-a49d-52cefe98ee71/volumes"
Nov 03 12:19:34.575982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.575963    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" path="/var/lib/kubelet/pods/dc0e3ad4-23fd-4535-bef5-a2a475c52d95/volumes"
Nov 03 12:19:34.808428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.808339    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-7c4a2b"]
Nov 03 12:19:34.809222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809197    4546 cpu_manager.go:414] "containerMap: removing container" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy"
Nov 03 12:19:34.809222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809221    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy"
Nov 03 12:19:34.809222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809239    4546 cpu_manager.go:414] "containerMap: removing container" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy"
Nov 03 12:19:34.809222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809246    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy"
Nov 03 12:19:34.809222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809269    4546 cpu_manager.go:414] "containerMap: removing container" podUID="93f10df6-086e-475b-bbd5-624329ba890e" containerName="payload-pre-processing"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809277    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f10df6-086e-475b-bbd5-624329ba890e" containerName="payload-pre-processing"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809291    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809298    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809308    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2a533ac7-f535-4725-bb9d-c7f3a76f44fd" containerName="payload-processing"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809316    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a533ac7-f535-4725-bb9d-c7f3a76f44fd" containerName="payload-processing"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809330    4546 cpu_manager.go:414] "containerMap: removing container" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809335    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809456    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809468    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0e3ad4-23fd-4535-bef5-a2a475c52d95" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809474    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fec5de-3713-448e-a3aa-18c43d550178" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809482    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f10df6-086e-475b-bbd5-624329ba890e" containerName="payload-pre-processing"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809488    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed6b079-a27d-4a6d-a49d-52cefe98ee71" containerName="istio-proxy"
Nov 03 12:19:34.809556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.809493    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a533ac7-f535-4725-bb9d-c7f3a76f44fd" containerName="payload-processing"
Nov 03 12:19:34.815215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.815195    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:34.850181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.850146    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cp4v\" (UniqueName: \"kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v\") pod \"test-curl-7593-7c4a2b\" (UID: \"fd2836e7-df13-4366-a241-c80b0e36b098\") " pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:34.863185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.863157    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-7c4a2b"]
Nov 03 12:19:34.863398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.863381    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ghw4\""
Nov 03 12:19:34.951797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:34.951712    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cp4v\" (UniqueName: \"kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v\") pod \"test-curl-7593-7c4a2b\" (UID: \"fd2836e7-df13-4366-a241-c80b0e36b098\") " pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:35.004960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:35.004932    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cp4v\" (UniqueName: \"kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v\") pod \"test-curl-7593-7c4a2b\" (UID: \"fd2836e7-df13-4366-a241-c80b0e36b098\") " pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:35.131739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:35.131660    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:35.493668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:35.493641    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-7c4a2b"]
Nov 03 12:19:35.494894 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:35.494869    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2836e7_df13_4366_a241_c80b0e36b098.slice/crio-fb45bb4caff2cc4b5293569408c728a78adfa3d4c3c827c37179cb9b0ac735aa WatchSource:0}: Error finding container fb45bb4caff2cc4b5293569408c728a78adfa3d4c3c827c37179cb9b0ac735aa: Status 404 returned error can't find the container with id fb45bb4caff2cc4b5293569408c728a78adfa3d4c3c827c37179cb9b0ac735aa
Nov 03 12:19:35.496876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:35.496860    4546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:19:35.796742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:35.796660    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-7c4a2b" event={"ID":"fd2836e7-df13-4366-a241-c80b0e36b098","Type":"ContainerStarted","Data":"fb45bb4caff2cc4b5293569408c728a78adfa3d4c3c827c37179cb9b0ac735aa"}
Nov 03 12:19:37.657201 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:37.657164    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:19:38.821220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:38.821185    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-7c4a2b" event={"ID":"fd2836e7-df13-4366-a241-c80b0e36b098","Type":"ContainerStarted","Data":"ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247"}
Nov 03 12:19:38.927689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:38.927642    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7593-7c4a2b" podStartSLOduration=2.026474519 podStartE2EDuration="4.92762822s" totalImagesPullingTime="2.901153701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:38.925185298 +0000 UTC m=+1720.496769208" watchObservedRunningTime="2026-11-03 12:19:38.92762822 +0000 UTC m=+1720.499212131"
Nov 03 12:19:40.340100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:40.340068    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-7c4a2b"]
Nov 03 12:19:40.344461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:40.344433    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-7c4a2b"]
Nov 03 12:19:40.835045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:40.835001    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7593-7c4a2b" podUID="fd2836e7-df13-4366-a241-c80b0e36b098" containerName="test-curl-7593-7c4a2b" containerID="cri-o://ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247" gracePeriod=2
Nov 03 12:19:42.243369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.243333    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"]
Nov 03 12:19:42.243928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.243914    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fd2836e7-df13-4366-a241-c80b0e36b098" containerName="test-curl-7593-7c4a2b"
Nov 03 12:19:42.243978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.243929    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2836e7-df13-4366-a241-c80b0e36b098" containerName="test-curl-7593-7c4a2b"
Nov 03 12:19:42.243978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.243953    4546 cpu_manager.go:414] "containerMap: removing container" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy"
Nov 03 12:19:42.243978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.243958    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy"
Nov 03 12:19:42.244077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.244044    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="511d427a-a19d-411f-9c7c-2fd4674f2c73" containerName="istio-proxy"
Nov 03 12:19:42.244077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.244057    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2836e7-df13-4366-a241-c80b0e36b098" containerName="test-curl-7593-7c4a2b"
Nov 03 12:19:42.247069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.247053    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.336612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.336578    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"]
Nov 03 12:19:42.345586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.345556    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.345764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.345602    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.345842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.345804    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.345894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.345843    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lfn\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.346026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.345989    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.346161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.346059    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.346161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.346134    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.346282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.346223    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.346282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.346255    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.447923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.447899    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.447960    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.447980    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448034    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448049    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lfn\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448113    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448153    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448219    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448265    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448357    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448497    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448565    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448706 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448679    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.448870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.448785    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.450541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.450518    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.450871 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.450851    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.500552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.500498    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.500714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.500693    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lfn\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.560631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.560606    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:42.980230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.980201    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"]
Nov 03 12:19:42.980803 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:42.980773    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcb9967_a55d_4625_9b2f_c9af04c7ef2a.slice/crio-8235dd3b7d31840bad4555d7b9f49cb741ae3bd6c6394b2d577bb08f5c40d98b WatchSource:0}: Error finding container 8235dd3b7d31840bad4555d7b9f49cb741ae3bd6c6394b2d577bb08f5c40d98b: Status 404 returned error can't find the container with id 8235dd3b7d31840bad4555d7b9f49cb741ae3bd6c6394b2d577bb08f5c40d98b
Nov 03 12:19:42.983580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.983546    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:42.983684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.983608    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:42.983684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:42.983637    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:19:43.033737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.033706    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:43.157697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.157625    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v\" (UniqueName: \"kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v\") pod \"fd2836e7-df13-4366-a241-c80b0e36b098\" (UID: \"fd2836e7-df13-4366-a241-c80b0e36b098\") "
Nov 03 12:19:43.164925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.164892    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v" pod "fd2836e7-df13-4366-a241-c80b0e36b098" (UID: "fd2836e7-df13-4366-a241-c80b0e36b098"). InnerVolumeSpecName "kube-api-access-2cp4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:43.259822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.259788    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cp4v\" (UniqueName: \"kubernetes.io/projected/fd2836e7-df13-4366-a241-c80b0e36b098-kube-api-access-2cp4v\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:43.856007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.855915    4546 generic.go:356] "Generic (PLEG): container finished" podID="fd2836e7-df13-4366-a241-c80b0e36b098" containerID="ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247" exitCode=137
Nov 03 12:19:43.856007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.855961    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:43.856239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.856018    4546 scope.go:122] "RemoveContainer" containerID="ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247"
Nov 03 12:19:43.858011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.857986    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" event={"ID":"adcb9967-a55d-4625-9b2f-c9af04c7ef2a","Type":"ContainerStarted","Data":"8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e"}
Nov 03 12:19:43.858151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.858041    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" event={"ID":"adcb9967-a55d-4625-9b2f-c9af04c7ef2a","Type":"ContainerStarted","Data":"8235dd3b7d31840bad4555d7b9f49cb741ae3bd6c6394b2d577bb08f5c40d98b"}
Nov 03 12:19:43.866384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.866368    4546 scope.go:122] "RemoveContainer" containerID="ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247"
Nov 03 12:19:43.866599 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:43.866577    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247\": container with ID starting with ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247 not found: ID does not exist" containerID="ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247"
Nov 03 12:19:43.866661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:43.866609    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247"} err="failed to get container status \"ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247\": rpc error: code = NotFound desc = could not find container \"ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247\": container with ID starting with ce30157ac953f014c577243c2eba20ad5b877cd54e532982a8ca35a4c7671247 not found: ID does not exist"
Nov 03 12:19:44.030773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:44.025368    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podStartSLOduration=2.025339065 podStartE2EDuration="2.025339065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:44.023067955 +0000 UTC m=+1725.594651882" watchObservedRunningTime="2026-11-03 12:19:44.025339065 +0000 UTC m=+1725.596922975"
Nov 03 12:19:44.070091 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:19:44.070050    4546 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-7c4a2b\" is forbidden: User \"system:node:ip-10-0-136-245.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-245.us-west-2.compute.internal' and this object" podUID="fd2836e7-df13-4366-a241-c80b0e36b098" pod="opendatahub/test-curl-7593-7c4a2b"
Nov 03 12:19:44.561492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:44.561461    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:44.563120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:44.563093    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 12:19:44.563248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:44.563137    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 12:19:44.573620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:44.573596    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2836e7-df13-4366-a241-c80b0e36b098" path="/var/lib/kubelet/pods/fd2836e7-df13-4366-a241-c80b0e36b098/volumes"
Nov 03 12:19:45.561302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:45.561271    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 12:19:45.561558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:45.561316    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 12:19:45.836528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:45.836434    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"]
Nov 03 12:19:46.561157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:46.561123    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 12:19:46.561351 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:46.561188    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 12:19:47.562264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:47.562225    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 12:19:47.562726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:47.562286    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 12:19:48.561170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:48.561126    4546 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 12:19:48.561335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:48.561182    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 12:19:49.565382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:49.565353    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:49.565797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:49.565784    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:49.566519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:49.566500    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:19:59.197220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.197190    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"]
Nov 03 12:19:59.202854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.202833    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.254849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.254792    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"]
Nov 03 12:19:59.297062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.297032    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e8d12\""
Nov 03 12:19:59.297226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.297142    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e8d12-dockercfg-kw5mh\""
Nov 03 12:19:59.361358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.361325    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr6ds\" (UniqueName: \"kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.361566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.361457    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.462780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.462674    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr6ds\" (UniqueName: \"kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.462975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.462844    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.463521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.463498    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.534245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.534219    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr6ds\" (UniqueName: \"kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds\") pod \"payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.541035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.541012    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:19:59.733009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.732979    4546 scope.go:122] "RemoveContainer" containerID="bf9160b9c58e008f8feaf67b2086973790a032d7c84a2c47bd7f62c6123a01de"
Nov 03 12:19:59.745956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.745933    4546 scope.go:122] "RemoveContainer" containerID="438a23862f6558b380384554fb03c9da75efac68dd4b76e1e73b792e135c8b98"
Nov 03 12:19:59.755495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.755472    4546 scope.go:122] "RemoveContainer" containerID="d22446977c0ac9e8bf893d8006d7a7dfa8440ef9df356a9919978db64cdd00d7"
Nov 03 12:19:59.764150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.764134    4546 scope.go:122] "RemoveContainer" containerID="0557233be27c8f274226d825eeff4539028ef8ae74dbe356c956f21705a94e38"
Nov 03 12:19:59.772884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.772849    4546 scope.go:122] "RemoveContainer" containerID="e242c5792ad881231276411f4b7b7f8f44c344e0847a392c92abdd4111f6d8e0"
Nov 03 12:19:59.781219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.781203    4546 scope.go:122] "RemoveContainer" containerID="a3393b76984f34c016bb6241306edc52861fc23b3cea5c9954ddb622a3415173"
Nov 03 12:19:59.889470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.889348    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"]
Nov 03 12:19:59.895458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.895443    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:19:59.935604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.935567    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"]
Nov 03 12:19:59.946444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.946323    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"]
Nov 03 12:19:59.950142 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:19:59.950116    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48fc8e4_7945_45eb_b810_281dfad016dd.slice/crio-f0305cdf39dcb578e66332091858ceb7b35b76424fb38bfe53b28878083d5575 WatchSource:0}: Error finding container f0305cdf39dcb578e66332091858ceb7b35b76424fb38bfe53b28878083d5575: Status 404 returned error can't find the container with id f0305cdf39dcb578e66332091858ceb7b35b76424fb38bfe53b28878083d5575
Nov 03 12:19:59.981186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:19:59.981142    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4" event={"ID":"b48fc8e4-7945-45eb-b810-281dfad016dd","Type":"ContainerStarted","Data":"f0305cdf39dcb578e66332091858ceb7b35b76424fb38bfe53b28878083d5575"}
Nov 03 12:20:00.071175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.071086    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.071332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.071195    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkpv\" (UniqueName: \"kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv\") pod \"payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.172666 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.172628    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.172892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.172714    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkpv\" (UniqueName: \"kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv\") pod \"payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.173406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.173385    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.232672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.232632    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkpv\" (UniqueName: \"kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv\") pod \"payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.242158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.242130    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:00.649768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.649679    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"]
Nov 03 12:20:00.857726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.857686    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r" podUID="74067bff-7993-408c-b816-31ce5428618c" containerName="main" containerID="cri-o://b98494b8e8222e7a9e56a067c09d11501639843356ed9a50a59614737b05bfdd" gracePeriod=45
Nov 03 12:20:00.990936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.990884    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g" event={"ID":"33b9d6fa-d74a-40d2-a7b0-853710870c0f","Type":"ContainerStarted","Data":"3bd86e430756b6fc168653ed0b086fe18caac221b2da32f15b3a908d7caca282"}
Nov 03 12:20:00.992691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.992663    4546 generic.go:356] "Generic (PLEG): container finished" podID="74067bff-7993-408c-b816-31ce5428618c" containerID="b98494b8e8222e7a9e56a067c09d11501639843356ed9a50a59614737b05bfdd" exitCode=0
Nov 03 12:20:00.992851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:00.992743    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r" event={"ID":"74067bff-7993-408c-b816-31ce5428618c","Type":"ContainerDied","Data":"b98494b8e8222e7a9e56a067c09d11501639843356ed9a50a59614737b05bfdd"}
Nov 03 12:20:01.168380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.168356    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:20:01.285365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285341    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs\" (UniqueName: \"kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs\") pod \"74067bff-7993-408c-b816-31ce5428618c\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") "
Nov 03 12:20:01.285831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285495    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6\" (UniqueName: \"kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6\") pod \"74067bff-7993-408c-b816-31ce5428618c\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") "
Nov 03 12:20:01.285831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285568    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache\") pod \"74067bff-7993-408c-b816-31ce5428618c\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") "
Nov 03 12:20:01.285831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285615    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm\") pod \"74067bff-7993-408c-b816-31ce5428618c\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") "
Nov 03 12:20:01.285831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285648    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home\") pod \"74067bff-7993-408c-b816-31ce5428618c\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") "
Nov 03 12:20:01.285831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285684    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir\") pod \"74067bff-7993-408c-b816-31ce5428618c\" (UID: \"74067bff-7993-408c-b816-31ce5428618c\") "
Nov 03 12:20:01.286141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.285898    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache" pod "74067bff-7993-408c-b816-31ce5428618c" (UID: "74067bff-7993-408c-b816-31ce5428618c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.286141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.286119    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home" pod "74067bff-7993-408c-b816-31ce5428618c" (UID: "74067bff-7993-408c-b816-31ce5428618c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.286295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.286278    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir" pod "74067bff-7993-408c-b816-31ce5428618c" (UID: "74067bff-7993-408c-b816-31ce5428618c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.286642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.286620    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.286698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.286647    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.286698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.286664    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.288148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.288123    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6" pod "74067bff-7993-408c-b816-31ce5428618c" (UID: "74067bff-7993-408c-b816-31ce5428618c"). InnerVolumeSpecName "kube-api-access-5qwk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:01.288535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.288518    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs" pod "74067bff-7993-408c-b816-31ce5428618c" (UID: "74067bff-7993-408c-b816-31ce5428618c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:20:01.289470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.289444    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm" pod "74067bff-7993-408c-b816-31ce5428618c" (UID: "74067bff-7993-408c-b816-31ce5428618c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:20:01.388715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.388627    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qwk6\" (UniqueName: \"kubernetes.io/projected/74067bff-7993-408c-b816-31ce5428618c-kube-api-access-5qwk6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.388715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.388673    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74067bff-7993-408c-b816-31ce5428618c-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:01.388715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:01.388689    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74067bff-7993-408c-b816-31ce5428618c-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:02.004022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:02.003995    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"
Nov 03 12:20:02.004224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:02.004030    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r" event={"ID":"74067bff-7993-408c-b816-31ce5428618c","Type":"ContainerDied","Data":"51188bea38449a1647a5cdbc4f4b9893de6f7a4a47eed847791212beff380318"}
Nov 03 12:20:02.004224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:02.004084    4546 scope.go:122] "RemoveContainer" containerID="b98494b8e8222e7a9e56a067c09d11501639843356ed9a50a59614737b05bfdd"
Nov 03 12:20:02.175783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:02.174253    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"]
Nov 03 12:20:02.240167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:02.240131    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-69fd8043/e2e-lifecycle-model-69fd8043-kserve-8f97b6789-tqz8r"]
Nov 03 12:20:02.603692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:02.603662    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74067bff-7993-408c-b816-31ce5428618c" path="/var/lib/kubelet/pods/74067bff-7993-408c-b816-31ce5428618c/volumes"
Nov 03 12:20:05.052489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:05.052419    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4" event={"ID":"b48fc8e4-7945-45eb-b810-281dfad016dd","Type":"ContainerStarted","Data":"b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a"}
Nov 03 12:20:05.054435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:05.054414    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:20:05.056889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:05.056842    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g" event={"ID":"33b9d6fa-d74a-40d2-a7b0-853710870c0f","Type":"ContainerStarted","Data":"16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4"}
Nov 03 12:20:05.057617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:05.057594    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:05.162868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:05.162819    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4" podStartSLOduration=1.796794503 podStartE2EDuration="6.162807351s" totalImagesPullingTime="4.366012848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:05.15995484 +0000 UTC m=+1746.731538751" watchObservedRunningTime="2026-11-03 12:20:05.162807351 +0000 UTC m=+1746.734391261"
Nov 03 12:20:05.270632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:05.270573    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g" podStartSLOduration=2.511134754 podStartE2EDuration="6.270557357s" totalImagesPullingTime="3.759422603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:05.268497188 +0000 UTC m=+1746.840081128" watchObservedRunningTime="2026-11-03 12:20:05.270557357 +0000 UTC m=+1746.842141267"
Nov 03 12:20:07.070872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:07.070841    4546 generic.go:356] "Generic (PLEG): container finished" podID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerID="07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad" exitCode=0
Nov 03 12:20:07.071380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:07.070918    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" event={"ID":"d52bbfaa-1be7-40e7-9caa-997651d7155c","Type":"ContainerDied","Data":"07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"}
Nov 03 12:20:07.071380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:07.071368    4546 scope.go:122] "RemoveContainer" containerID="07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"
Nov 03 12:20:07.072974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:07.072948    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:07.073056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:07.073012    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:07.073056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:07.073046    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:07.698354 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:20:07.698279    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:20:08.079344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:08.079310    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" event={"ID":"d52bbfaa-1be7-40e7-9caa-997651d7155c","Type":"ContainerStarted","Data":"1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7"}
Nov 03 12:20:08.268222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:08.268185    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:20:08.268432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:08.268342    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:20:08.268432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:08.268410    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 12:20:09.268149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:09.268111    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:20:09.268558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:09.268170    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 12:20:10.268186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:10.268148    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:20:10.268714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:10.268222    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 12:20:11.268965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.268935    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:20:11.269408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.269020    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 12:20:11.854204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.854169    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"]
Nov 03 12:20:11.854902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.854882    4546 cpu_manager.go:414] "containerMap: removing container" podUID="74067bff-7993-408c-b816-31ce5428618c" containerName="main"
Nov 03 12:20:11.855002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.854904    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74067bff-7993-408c-b816-31ce5428618c" containerName="main"
Nov 03 12:20:11.855046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.855029    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="74067bff-7993-408c-b816-31ce5428618c" containerName="main"
Nov 03 12:20:11.862952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.862933    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873359    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873400    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873488    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873525    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873591    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873672    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873689    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdz9b\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873806    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.873862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.873826    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.924839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.924793    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2f3eb2-openshift-default-dockercfg-qz2hr\""
Nov 03 12:20:11.975583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975531    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975601    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975660    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975801    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975825    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdz9b\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975894    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975920    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.975994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.975978    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.976040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.976025    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.976521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.976495    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.976708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.976641    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.976890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.976865    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.977591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.977547    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"]
Nov 03 12:20:11.977978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.977956    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.978302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.978279    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.980236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.980067    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:11.980236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:11.980210    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:12.060788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.060189    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:12.060788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.060424    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdz9b\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b\") pod \"e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") " pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:12.176572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.176446    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:12.268348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.268323    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 12:20:12.268477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.268377    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 12:20:12.638146 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:20:12.638103    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f3b83c_ac0b_47c3_93a5_441f9d349054.slice/crio-3bee9afc82de8a54a36984e786d971b99f8eb4ac2fd6f081c35bc41b0311942d WatchSource:0}: Error finding container 3bee9afc82de8a54a36984e786d971b99f8eb4ac2fd6f081c35bc41b0311942d: Status 404 returned error can't find the container with id 3bee9afc82de8a54a36984e786d971b99f8eb4ac2fd6f081c35bc41b0311942d
Nov 03 12:20:12.638652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.638627    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"]
Nov 03 12:20:12.640818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.640783    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:12.640937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.640872    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:12.640937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:12.640917    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:13.173801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.173729    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" event={"ID":"b2f3b83c-ac0b-47c3-93a5-441f9d349054","Type":"ContainerStarted","Data":"7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"}
Nov 03 12:20:13.173993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.173827    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" event={"ID":"b2f3b83c-ac0b-47c3-93a5-441f9d349054","Type":"ContainerStarted","Data":"3bee9afc82de8a54a36984e786d971b99f8eb4ac2fd6f081c35bc41b0311942d"}
Nov 03 12:20:13.177202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.177182    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:13.183840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.183811    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:13.285880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.285826    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podStartSLOduration=2.2858119820000002 podStartE2EDuration="2.285811982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:13.281388774 +0000 UTC m=+1754.852972689" watchObservedRunningTime="2026-11-03 12:20:13.285811982 +0000 UTC m=+1754.857395894"
Nov 03 12:20:13.361592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.361554    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:20:13.362057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.362027    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:20:13.363165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:13.363134    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:20:14.180442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:14.180412    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:14.181437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:14.181414    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:20:17.073958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:17.073915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:20:17.074506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:17.074345    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:20:19.442881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.442832    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"]
Nov 03 12:20:19.447698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.447677    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.503285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.503244    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"]
Nov 03 12:20:19.543416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.543383    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2f3eb2\""
Nov 03 12:20:19.543915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.543893    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2f3eb2-dockercfg-xsfpp\""
Nov 03 12:20:19.646583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.628793    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.646941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.646908    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29s7\" (UniqueName: \"kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.657806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.648569    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"]
Nov 03 12:20:19.660051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.660029    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.674381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.674356    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"]
Nov 03 12:20:19.752868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.752831    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqtg\" (UniqueName: \"kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg\") pod \"payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.753059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.752890    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.753151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.753125    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.753190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.753178    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29s7\" (UniqueName: \"kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.753870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.753852    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.808503 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.808472    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29s7\" (UniqueName: \"kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:19.854146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.854120    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqtg\" (UniqueName: \"kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg\") pod \"payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.854304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.854175    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.854818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.854800    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.907030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.906999    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqtg\" (UniqueName: \"kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg\") pod \"payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:19.977859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:19.977823    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:20.062802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:20.061565    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:20.352438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:20.352410    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"]
Nov 03 12:20:20.352919 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:20:20.352892    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81a81b7_2c19_423d_8a17_fa7e44c926db.slice/crio-4a1319234c46bcbe84a432d33f5c42a767e1d6d45cdf3f01266005fa62cc5bc8 WatchSource:0}: Error finding container 4a1319234c46bcbe84a432d33f5c42a767e1d6d45cdf3f01266005fa62cc5bc8: Status 404 returned error can't find the container with id 4a1319234c46bcbe84a432d33f5c42a767e1d6d45cdf3f01266005fa62cc5bc8
Nov 03 12:20:20.445274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:20.445248    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"]
Nov 03 12:20:20.446381 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:20:20.446351    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1fe0d7_3764_42a6_bc17_0d8d18bd9b48.slice/crio-fb103340eef76110a63e7d6c03ef5da0db5e2c85b09ead6191d8f3232bd554ce WatchSource:0}: Error finding container fb103340eef76110a63e7d6c03ef5da0db5e2c85b09ead6191d8f3232bd554ce: Status 404 returned error can't find the container with id fb103340eef76110a63e7d6c03ef5da0db5e2c85b09ead6191d8f3232bd554ce
Nov 03 12:20:21.227213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.227175    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j" event={"ID":"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48","Type":"ContainerStarted","Data":"2394fd69dde59829933fcffd25e57be8037208725ad4edb78734e2efdab0f5bf"}
Nov 03 12:20:21.227433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.227236    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j" event={"ID":"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48","Type":"ContainerStarted","Data":"fb103340eef76110a63e7d6c03ef5da0db5e2c85b09ead6191d8f3232bd554ce"}
Nov 03 12:20:21.227433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.227269    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:21.229015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.228988    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq" event={"ID":"f81a81b7-2c19-423d-8a17-fa7e44c926db","Type":"ContainerStarted","Data":"320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56"}
Nov 03 12:20:21.229140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.229032    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq" event={"ID":"f81a81b7-2c19-423d-8a17-fa7e44c926db","Type":"ContainerStarted","Data":"4a1319234c46bcbe84a432d33f5c42a767e1d6d45cdf3f01266005fa62cc5bc8"}
Nov 03 12:20:21.229140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.229054    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:21.333553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.333498    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j" podStartSLOduration=2.33348427 podStartE2EDuration="2.33348427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:21.331901143 +0000 UTC m=+1762.903485054" watchObservedRunningTime="2026-11-03 12:20:21.33348427 +0000 UTC m=+1762.905068181"
Nov 03 12:20:21.469507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:21.469446    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq" podStartSLOduration=2.46942829 podStartE2EDuration="2.46942829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:21.465321051 +0000 UTC m=+1763.036905006" watchObservedRunningTime="2026-11-03 12:20:21.46942829 +0000 UTC m=+1763.041012202"
Nov 03 12:20:24.125048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:24.125015    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"]
Nov 03 12:20:29.130776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.129819    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb" podUID="bfd17a3b-6147-4474-ad8b-84079199bacb" containerName="payload-pre-processing" containerID="cri-o://f7aaf62148cb15ebc24606d6c091e95f91ec456f9874eccd79fa83d77a347cd9" gracePeriod=25
Nov 03 12:20:29.305291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.305258    4546 generic.go:356] "Generic (PLEG): container finished" podID="bfd17a3b-6147-4474-ad8b-84079199bacb" containerID="f7aaf62148cb15ebc24606d6c091e95f91ec456f9874eccd79fa83d77a347cd9" exitCode=0
Nov 03 12:20:29.305494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.305324    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb" event={"ID":"bfd17a3b-6147-4474-ad8b-84079199bacb","Type":"ContainerDied","Data":"f7aaf62148cb15ebc24606d6c091e95f91ec456f9874eccd79fa83d77a347cd9"}
Nov 03 12:20:29.453215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.453183    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:20:29.551019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.550976    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx\") pod \"bfd17a3b-6147-4474-ad8b-84079199bacb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") "
Nov 03 12:20:29.551240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.551064    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume\") pod \"bfd17a3b-6147-4474-ad8b-84079199bacb\" (UID: \"bfd17a3b-6147-4474-ad8b-84079199bacb\") "
Nov 03 12:20:29.551544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.551517    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume" pod "bfd17a3b-6147-4474-ad8b-84079199bacb" (UID: "bfd17a3b-6147-4474-ad8b-84079199bacb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:29.551782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.551741    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd17a3b-6147-4474-ad8b-84079199bacb-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:29.553763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.553724    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx" pod "bfd17a3b-6147-4474-ad8b-84079199bacb" (UID: "bfd17a3b-6147-4474-ad8b-84079199bacb"). InnerVolumeSpecName "kube-api-access-ztrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:29.653250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:29.653161    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/bfd17a3b-6147-4474-ad8b-84079199bacb-kube-api-access-ztrpx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:30.333710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:30.333663    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb" event={"ID":"bfd17a3b-6147-4474-ad8b-84079199bacb","Type":"ContainerDied","Data":"6e7a7f9f98493883ace8072686687c6a5b80b5f20291b38ff82eb8faa6b41550"}
Nov 03 12:20:30.333710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:30.333695    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"
Nov 03 12:20:30.334260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:30.333833    4546 scope.go:122] "RemoveContainer" containerID="f7aaf62148cb15ebc24606d6c091e95f91ec456f9874eccd79fa83d77a347cd9"
Nov 03 12:20:30.488957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:30.488920    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"]
Nov 03 12:20:30.492576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:30.492541    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69fd8043-d6457c4bc-44hwb"]
Nov 03 12:20:30.574661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:30.574630    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd17a3b-6147-4474-ad8b-84079199bacb" path="/var/lib/kubelet/pods/bfd17a3b-6147-4474-ad8b-84079199bacb/volumes"
Nov 03 12:20:32.239423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:32.239385    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:20:32.239966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:32.239458    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:20:34.364053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:34.363970    4546 generic.go:356] "Generic (PLEG): container finished" podID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerID="a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1" exitCode=0
Nov 03 12:20:34.364575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:34.364046    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" event={"ID":"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6","Type":"ContainerDied","Data":"a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"}
Nov 03 12:20:34.364640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:34.364603    4546 scope.go:122] "RemoveContainer" containerID="a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"
Nov 03 12:20:34.366268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:34.366241    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:34.366363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:34.366307    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:34.366363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:34.366341    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:35.372463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:35.372428    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" event={"ID":"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6","Type":"ContainerStarted","Data":"abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6"}
Nov 03 12:20:36.092051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:36.092015    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:20:36.093093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:36.093062    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 12:20:36.093220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:36.093114    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 12:20:37.092527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:37.092496    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 12:20:37.092986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:37.092548    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 12:20:37.662219 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:20:37.662179    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:20:38.092869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:38.092832    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 12:20:38.093304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:38.092898    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 12:20:38.227776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:38.225873    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:20:39.092459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:39.092423    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 12:20:39.092681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:39.092487    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 12:20:40.092372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:40.092335    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 12:20:40.092829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:40.092394    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 12:20:41.097619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:41.097584    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:20:41.098698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:41.098677    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:20:41.608511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:41.608479    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"]
Nov 03 12:20:41.702983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:41.702945    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"]
Nov 03 12:20:46.608809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:46.608769    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4" podUID="08543efc-bb86-4e93-bd36-57f09f0f02af" containerName="payload-processing" containerID="cri-o://73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb" gracePeriod=25
Nov 03 12:20:46.704209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:46.704171    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l" podUID="ebe83552-cbbd-44bd-a3fc-b237f8f486c3" containerName="payload-pre-processing" containerID="cri-o://c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989" gracePeriod=25
Nov 03 12:20:47.002764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.002723    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:20:47.036972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.036940    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:20:47.105917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.105881    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r\" (UniqueName: \"kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r\") pod \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") "
Nov 03 12:20:47.106097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.105964    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume\") pod \"08543efc-bb86-4e93-bd36-57f09f0f02af\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") "
Nov 03 12:20:47.106180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.106098    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume\") pod \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\" (UID: \"ebe83552-cbbd-44bd-a3fc-b237f8f486c3\") "
Nov 03 12:20:47.106241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.106227    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv\" (UniqueName: \"kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv\") pod \"08543efc-bb86-4e93-bd36-57f09f0f02af\" (UID: \"08543efc-bb86-4e93-bd36-57f09f0f02af\") "
Nov 03 12:20:47.106438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.106414    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume" pod "08543efc-bb86-4e93-bd36-57f09f0f02af" (UID: "08543efc-bb86-4e93-bd36-57f09f0f02af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:47.106505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.106460    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume" pod "ebe83552-cbbd-44bd-a3fc-b237f8f486c3" (UID: "ebe83552-cbbd-44bd-a3fc-b237f8f486c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:47.106785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.106764    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08543efc-bb86-4e93-bd36-57f09f0f02af-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:47.106854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.106788    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:47.108655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.108625    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r" pod "ebe83552-cbbd-44bd-a3fc-b237f8f486c3" (UID: "ebe83552-cbbd-44bd-a3fc-b237f8f486c3"). InnerVolumeSpecName "kube-api-access-hld2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:47.108779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.108709    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv" pod "08543efc-bb86-4e93-bd36-57f09f0f02af" (UID: "08543efc-bb86-4e93-bd36-57f09f0f02af"). InnerVolumeSpecName "kube-api-access-6p5cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:47.207925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.207841    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5cv\" (UniqueName: \"kubernetes.io/projected/08543efc-bb86-4e93-bd36-57f09f0f02af-kube-api-access-6p5cv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:47.207925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.207870    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hld2r\" (UniqueName: \"kubernetes.io/projected/ebe83552-cbbd-44bd-a3fc-b237f8f486c3-kube-api-access-hld2r\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:47.469437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.469399    4546 generic.go:356] "Generic (PLEG): container finished" podID="ebe83552-cbbd-44bd-a3fc-b237f8f486c3" containerID="c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989" exitCode=0
Nov 03 12:20:47.469642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.469454    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"
Nov 03 12:20:47.469642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.469472    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l" event={"ID":"ebe83552-cbbd-44bd-a3fc-b237f8f486c3","Type":"ContainerDied","Data":"c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989"}
Nov 03 12:20:47.469642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.469573    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l" event={"ID":"ebe83552-cbbd-44bd-a3fc-b237f8f486c3","Type":"ContainerDied","Data":"b105730de19d3a73effcc6329d91ffa97b833a1aff81fab53b7c79da551c9e53"}
Nov 03 12:20:47.469642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.469606    4546 scope.go:122] "RemoveContainer" containerID="c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989"
Nov 03 12:20:47.471077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.471054    4546 generic.go:356] "Generic (PLEG): container finished" podID="08543efc-bb86-4e93-bd36-57f09f0f02af" containerID="73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb" exitCode=0
Nov 03 12:20:47.471188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.471115    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4" event={"ID":"08543efc-bb86-4e93-bd36-57f09f0f02af","Type":"ContainerDied","Data":"73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb"}
Nov 03 12:20:47.471188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.471129    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"
Nov 03 12:20:47.471188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.471159    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4" event={"ID":"08543efc-bb86-4e93-bd36-57f09f0f02af","Type":"ContainerDied","Data":"a72ed71301b2bae0bc38601e5c886e9331df1267585011b13d7ea015f5b63f30"}
Nov 03 12:20:47.482490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.482463    4546 scope.go:122] "RemoveContainer" containerID="c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989"
Nov 03 12:20:47.482935 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:20:47.482902    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989\": container with ID starting with c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989 not found: ID does not exist" containerID="c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989"
Nov 03 12:20:47.483042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.482943    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989"} err="failed to get container status \"c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989\": rpc error: code = NotFound desc = could not find container \"c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989\": container with ID starting with c08773f5e66e098e9c21f784ae47caaa709e6a1fa0fa6ecb0974dbb7f4cdc989 not found: ID does not exist"
Nov 03 12:20:47.483042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.482967    4546 scope.go:122] "RemoveContainer" containerID="73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb"
Nov 03 12:20:47.493411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.493390    4546 scope.go:122] "RemoveContainer" containerID="73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb"
Nov 03 12:20:47.493737 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:20:47.493715    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb\": container with ID starting with 73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb not found: ID does not exist" containerID="73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb"
Nov 03 12:20:47.493832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.493780    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb"} err="failed to get container status \"73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb\": rpc error: code = NotFound desc = could not find container \"73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb\": container with ID starting with 73b19997698ca0c7c7b5874f1ac0f9ef6745bc332d4183a8709faa369178eefb not found: ID does not exist"
Nov 03 12:20:47.623479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.623448    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"]
Nov 03 12:20:47.670036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.670000    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-d6df894bc-rs98l"]
Nov 03 12:20:47.771341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.771240    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"]
Nov 03 12:20:47.774680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:47.774651    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7766b7f64c-vlgx4"]
Nov 03 12:20:48.573463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:48.573432    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08543efc-bb86-4e93-bd36-57f09f0f02af" path="/var/lib/kubelet/pods/08543efc-bb86-4e93-bd36-57f09f0f02af/volumes"
Nov 03 12:20:48.573893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:48.573880    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe83552-cbbd-44bd-a3fc-b237f8f486c3" path="/var/lib/kubelet/pods/ebe83552-cbbd-44bd-a3fc-b237f8f486c3/volumes"
Nov 03 12:20:53.569295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:53.569262    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"]
Nov 03 12:20:54.137695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.137655    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"]
Nov 03 12:20:54.138338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138320    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bfd17a3b-6147-4474-ad8b-84079199bacb" containerName="payload-pre-processing"
Nov 03 12:20:54.138398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138341    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd17a3b-6147-4474-ad8b-84079199bacb" containerName="payload-pre-processing"
Nov 03 12:20:54.138398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138359    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ebe83552-cbbd-44bd-a3fc-b237f8f486c3" containerName="payload-pre-processing"
Nov 03 12:20:54.138398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138365    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe83552-cbbd-44bd-a3fc-b237f8f486c3" containerName="payload-pre-processing"
Nov 03 12:20:54.138398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138393    4546 cpu_manager.go:414] "containerMap: removing container" podUID="08543efc-bb86-4e93-bd36-57f09f0f02af" containerName="payload-processing"
Nov 03 12:20:54.138398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138398    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08543efc-bb86-4e93-bd36-57f09f0f02af" containerName="payload-processing"
Nov 03 12:20:54.138561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138493    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="08543efc-bb86-4e93-bd36-57f09f0f02af" containerName="payload-processing"
Nov 03 12:20:54.138561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138506    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd17a3b-6147-4474-ad8b-84079199bacb" containerName="payload-pre-processing"
Nov 03 12:20:54.138561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.138513    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe83552-cbbd-44bd-a3fc-b237f8f486c3" containerName="payload-pre-processing"
Nov 03 12:20:54.142345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.142325    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.192692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.192661    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-583bcf-openshift-default-dockercfg-qbzgz\""
Nov 03 12:20:54.204883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.204846    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"]
Nov 03 12:20:54.322873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.322831    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.323213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.323190    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.323374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.323359    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.323578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.323562    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.324144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.324123    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7gf\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.324488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.324471    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.324652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.324637    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.324916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.324891    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.325131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.325112    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.426874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.426776    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.426874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.426823    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.426950    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427027    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427084    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427112    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427135    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427223    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427280    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7gf\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427559    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427663    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427699    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.427768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.427716    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.428222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.428205    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.429837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.429815    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.430170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.430151    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.479986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.479949    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.480161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.480088    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7gf\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf\") pod \"e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") " pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:54.756469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:54.756430    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:55.189966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.189869    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"]
Nov 03 12:20:55.223534 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:20:55.223490    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe14a45a_e6d6_4e34_915a_92fd4ba3e56f.slice/crio-4384a80267ca32919df2f1bc75e011d634b017c4a21ef127dad4f6579fba9fb1 WatchSource:0}: Error finding container 4384a80267ca32919df2f1bc75e011d634b017c4a21ef127dad4f6579fba9fb1: Status 404 returned error can't find the container with id 4384a80267ca32919df2f1bc75e011d634b017c4a21ef127dad4f6579fba9fb1
Nov 03 12:20:55.226510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.226468    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:55.226589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.226560    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:55.226638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.226604    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:20:55.527982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.527940    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" event={"ID":"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f","Type":"ContainerStarted","Data":"f36b9438b77c59ac0c3097abec447996adc816795a2c72e57169c4cf656d460c"}
Nov 03 12:20:55.528168 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.528008    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" event={"ID":"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f","Type":"ContainerStarted","Data":"4384a80267ca32919df2f1bc75e011d634b017c4a21ef127dad4f6579fba9fb1"}
Nov 03 12:20:55.639116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.639041    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podStartSLOduration=1.639021176 podStartE2EDuration="1.639021176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:55.636168354 +0000 UTC m=+1797.207752294" watchObservedRunningTime="2026-11-03 12:20:55.639021176 +0000 UTC m=+1797.210605089"
Nov 03 12:20:55.757357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:55.757254    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:56.764714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:56.764683    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:57.541812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:57.541777    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:57.542722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:57.542706    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:20:58.570173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.570125    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8" podUID="f702f631-1e55-4bb0-86ff-029906263672" containerName="payload-processing" containerID="cri-o://95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7" gracePeriod=25
Nov 03 12:20:58.860832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.860810    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:20:58.901101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.901077    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr\" (UniqueName: \"kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr\") pod \"f702f631-1e55-4bb0-86ff-029906263672\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") "
Nov 03 12:20:58.901230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.901164    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume\") pod \"f702f631-1e55-4bb0-86ff-029906263672\" (UID: \"f702f631-1e55-4bb0-86ff-029906263672\") "
Nov 03 12:20:58.901501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.901480    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume" pod "f702f631-1e55-4bb0-86ff-029906263672" (UID: "f702f631-1e55-4bb0-86ff-029906263672"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:58.901590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.901578    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702f631-1e55-4bb0-86ff-029906263672-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:58.903258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:58.903237    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr" pod "f702f631-1e55-4bb0-86ff-029906263672" (UID: "f702f631-1e55-4bb0-86ff-029906263672"). InnerVolumeSpecName "kube-api-access-dw5rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:59.003172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.003140    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5rr\" (UniqueName: \"kubernetes.io/projected/f702f631-1e55-4bb0-86ff-029906263672-kube-api-access-dw5rr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:59.557396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.557358    4546 generic.go:356] "Generic (PLEG): container finished" podID="f702f631-1e55-4bb0-86ff-029906263672" containerID="95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7" exitCode=0
Nov 03 12:20:59.557638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.557401    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"
Nov 03 12:20:59.557638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.557455    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8" event={"ID":"f702f631-1e55-4bb0-86ff-029906263672","Type":"ContainerDied","Data":"95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7"}
Nov 03 12:20:59.557638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.557546    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8" event={"ID":"f702f631-1e55-4bb0-86ff-029906263672","Type":"ContainerDied","Data":"799fb314095d43691d72bcdaf4f01c6c185beb2e37a4004553f361d4f62074b1"}
Nov 03 12:20:59.557638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.557579    4546 scope.go:122] "RemoveContainer" containerID="95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7"
Nov 03 12:20:59.568585 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.568568    4546 scope.go:122] "RemoveContainer" containerID="95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7"
Nov 03 12:20:59.568878 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:20:59.568857    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7\": container with ID starting with 95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7 not found: ID does not exist" containerID="95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7"
Nov 03 12:20:59.568959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.568885    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7"} err="failed to get container status \"95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7\": rpc error: code = NotFound desc = could not find container \"95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7\": container with ID starting with 95e59a777768cdf5b09e7b611e61f0b5654e64ec9e66ec10be9429bdd24e0be7 not found: ID does not exist"
Nov 03 12:20:59.710599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.710566    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"]
Nov 03 12:20:59.756582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.756551    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-c44dbb46f-wqtx8"]
Nov 03 12:20:59.908730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:20:59.908648    4546 scope.go:122] "RemoveContainer" containerID="658214e7761a0fada328df211225beb29522795b1001dd0299671ddb277ad40b"
Nov 03 12:21:00.572819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:00.572789    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f702f631-1e55-4bb0-86ff-029906263672" path="/var/lib/kubelet/pods/f702f631-1e55-4bb0-86ff-029906263672/volumes"
Nov 03 12:21:02.452381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.452349    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"]
Nov 03 12:21:02.453241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.453222    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f702f631-1e55-4bb0-86ff-029906263672" containerName="payload-processing"
Nov 03 12:21:02.453292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.453244    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f702f631-1e55-4bb0-86ff-029906263672" containerName="payload-processing"
Nov 03 12:21:02.453458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.453443    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f702f631-1e55-4bb0-86ff-029906263672" containerName="payload-processing"
Nov 03 12:21:02.459020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.458997    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.510847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.510821    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"]
Nov 03 12:21:02.548171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.548137    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8z8\" (UniqueName: \"kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.548444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.548420    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.553204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.553172    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-583bcf\""
Nov 03 12:21:02.553358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.553286    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-583bcf-dockercfg-hn4hf\""
Nov 03 12:21:02.655984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.655693    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8z8\" (UniqueName: \"kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.655984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.655942    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.670774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.656880    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.684772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.681620    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"]
Nov 03 12:21:02.700378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.695358    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:02.710827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.708144    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"]
Nov 03 12:21:02.754805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.754770    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8z8\" (UniqueName: \"kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.757462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.757432    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2m2n\" (UniqueName: \"kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n\") pod \"payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:02.757658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.757628    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:02.773662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.773637    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:02.859633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.859595    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:02.859824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.859730    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2m2n\" (UniqueName: \"kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n\") pod \"payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:02.860906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.860852    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:02.914203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:02.914170    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2m2n\" (UniqueName: \"kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n\") pod \"payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:03.076071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.076034    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:03.146264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.146151    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"]
Nov 03 12:21:03.448650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.448619    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"]
Nov 03 12:21:03.449233 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:03.449207    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4402fad5_d02e_4966_82fb_af42f96142b0.slice/crio-40d602bd9e500776cf9f680bf1b7f7e870730e56aff1d316a79121ea3bd48fb0 WatchSource:0}: Error finding container 40d602bd9e500776cf9f680bf1b7f7e870730e56aff1d316a79121ea3bd48fb0: Status 404 returned error can't find the container with id 40d602bd9e500776cf9f680bf1b7f7e870730e56aff1d316a79121ea3bd48fb0
Nov 03 12:21:03.600525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.600493    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt" event={"ID":"d282d4ac-1ef8-412b-9d54-e91ec08a25a1","Type":"ContainerStarted","Data":"ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4"}
Nov 03 12:21:03.600981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.600550    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt" event={"ID":"d282d4ac-1ef8-412b-9d54-e91ec08a25a1","Type":"ContainerStarted","Data":"24a934b78c20fc3958ff06dbb39c32ffb618ce06603dc71c0ac03b164a454841"}
Nov 03 12:21:03.600981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.600588    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:03.602243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.602221    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb" event={"ID":"4402fad5-d02e-4966-82fb-af42f96142b0","Type":"ContainerStarted","Data":"ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d"}
Nov 03 12:21:03.602370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.602264    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb" event={"ID":"4402fad5-d02e-4966-82fb-af42f96142b0","Type":"ContainerStarted","Data":"40d602bd9e500776cf9f680bf1b7f7e870730e56aff1d316a79121ea3bd48fb0"}
Nov 03 12:21:03.602370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.602291    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:03.706813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.706674    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt" podStartSLOduration=1.706660262 podStartE2EDuration="1.706660262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:03.706165031 +0000 UTC m=+1805.277748955" watchObservedRunningTime="2026-11-03 12:21:03.706660262 +0000 UTC m=+1805.278244208"
Nov 03 12:21:03.814365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:03.814308    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb" podStartSLOduration=1.8142925669999999 podStartE2EDuration="1.814292567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:03.811681756 +0000 UTC m=+1805.383265685" watchObservedRunningTime="2026-11-03 12:21:03.814292567 +0000 UTC m=+1805.385876506"
Nov 03 12:21:07.653285 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:21:07.653248    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:21:10.103246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:10.103203    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"]
Nov 03 12:21:10.201984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:10.201953    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"]
Nov 03 12:21:11.549557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.549519    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"]
Nov 03 12:21:11.554535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.554506    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.604303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.604264    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:21:11.604474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.604341    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:21:11.604474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.604433    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:21:11.604599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.604494    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:21:11.608866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.608813    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"]
Nov 03 12:21:11.695941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.695907    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8s7v\" (UniqueName: \"kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.696118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.696027    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.696194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.696173    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.797563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.797522    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8s7v\" (UniqueName: \"kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.797778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.797590    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.797778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.797679    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.798374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.798349    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.800929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.800870    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.851432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.851397    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8s7v\" (UniqueName: \"kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v\") pod \"maas-api-revoke-keys-e2e-mt-69fd8043-xngrg\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:11.871922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:11.871875    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:12.241219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:12.241189    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"]
Nov 03 12:21:12.241589 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:12.241567    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb02115_8e78_48fa_97a6_b3282da727ca.slice/crio-5127765d69b2bc02265a34453d99e2268020419dd2e6436d6c771e73bddde6ac WatchSource:0}: Error finding container 5127765d69b2bc02265a34453d99e2268020419dd2e6436d6c771e73bddde6ac: Status 404 returned error can't find the container with id 5127765d69b2bc02265a34453d99e2268020419dd2e6436d6c771e73bddde6ac
Nov 03 12:21:12.685772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:12.685710    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg" event={"ID":"3eb02115-8e78-48fa-97a6-b3282da727ca","Type":"ContainerStarted","Data":"52cbf5d5a6f33709fe221ebce8e07eb5d8df9b175a49e0770c63e3d071c3d227"}
Nov 03 12:21:12.685772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:12.685777    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg" event={"ID":"3eb02115-8e78-48fa-97a6-b3282da727ca","Type":"ContainerStarted","Data":"5127765d69b2bc02265a34453d99e2268020419dd2e6436d6c771e73bddde6ac"}
Nov 03 12:21:12.793334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:12.793225    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg" podStartSLOduration=1.7932141160000001 podStartE2EDuration="1.793214116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:12.792032546 +0000 UTC m=+1814.363616469" watchObservedRunningTime="2026-11-03 12:21:12.793214116 +0000 UTC m=+1814.364798027"
Nov 03 12:21:13.693771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:13.693716    4546 generic.go:356] "Generic (PLEG): container finished" podID="3eb02115-8e78-48fa-97a6-b3282da727ca" containerID="52cbf5d5a6f33709fe221ebce8e07eb5d8df9b175a49e0770c63e3d071c3d227" exitCode=0
Nov 03 12:21:13.694162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:13.693785    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg" event={"ID":"3eb02115-8e78-48fa-97a6-b3282da727ca","Type":"ContainerDied","Data":"52cbf5d5a6f33709fe221ebce8e07eb5d8df9b175a49e0770c63e3d071c3d227"}
Nov 03 12:21:14.613626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:14.613593    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:21:14.614024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:14.614001    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:21:14.887591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:14.887564    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:15.042000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.041966    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca\") pod \"3eb02115-8e78-48fa-97a6-b3282da727ca\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") "
Nov 03 12:21:15.042210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.042193    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v\" (UniqueName: \"kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v\") pod \"3eb02115-8e78-48fa-97a6-b3282da727ca\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") "
Nov 03 12:21:15.042300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.042277    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca\") pod \"3eb02115-8e78-48fa-97a6-b3282da727ca\" (UID: \"3eb02115-8e78-48fa-97a6-b3282da727ca\") "
Nov 03 12:21:15.042988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.042962    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca" pod "3eb02115-8e78-48fa-97a6-b3282da727ca" (UID: "3eb02115-8e78-48fa-97a6-b3282da727ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:21:15.044478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.044452    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca" pod "3eb02115-8e78-48fa-97a6-b3282da727ca" (UID: "3eb02115-8e78-48fa-97a6-b3282da727ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:21:15.045169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.045145    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v" pod "3eb02115-8e78-48fa-97a6-b3282da727ca" (UID: "3eb02115-8e78-48fa-97a6-b3282da727ca"). InnerVolumeSpecName "kube-api-access-v8s7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:21:15.104208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.104162    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8" podUID="73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" containerName="payload-processing" containerID="cri-o://a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5" gracePeriod=25
Nov 03 12:21:15.143691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.143648    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.143691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.143690    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8s7v\" (UniqueName: \"kubernetes.io/projected/3eb02115-8e78-48fa-97a6-b3282da727ca-kube-api-access-v8s7v\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.143691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.143701    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb02115-8e78-48fa-97a6-b3282da727ca-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.202647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.202592    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc" podUID="e6120c16-264d-4a54-b4a3-7ba38464dbb7" containerName="payload-pre-processing" containerID="cri-o://b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a" gracePeriod=25
Nov 03 12:21:15.367364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.367336    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:21:15.448297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.448265    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp\" (UniqueName: \"kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp\") pod \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") "
Nov 03 12:21:15.448513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.448422    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume\") pod \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\" (UID: \"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0\") "
Nov 03 12:21:15.449080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.449047    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume" pod "73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" (UID: "73c640c9-fcb6-4c15-bbbd-11787a1fa5f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:21:15.451329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.451290    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp" pod "73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" (UID: "73c640c9-fcb6-4c15-bbbd-11787a1fa5f0"). InnerVolumeSpecName "kube-api-access-2fwnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:21:15.459665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.459641    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:21:15.550464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.550415    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs\" (UniqueName: \"kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs\") pod \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") "
Nov 03 12:21:15.550677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.550482    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume\") pod \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\" (UID: \"e6120c16-264d-4a54-b4a3-7ba38464dbb7\") "
Nov 03 12:21:15.550990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.550960    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume" pod "e6120c16-264d-4a54-b4a3-7ba38464dbb7" (UID: "e6120c16-264d-4a54-b4a3-7ba38464dbb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:21:15.551115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.551030    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwnp\" (UniqueName: \"kubernetes.io/projected/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-kube-api-access-2fwnp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.551115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.551057    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.553054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.553027    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs" pod "e6120c16-264d-4a54-b4a3-7ba38464dbb7" (UID: "e6120c16-264d-4a54-b4a3-7ba38464dbb7"). InnerVolumeSpecName "kube-api-access-6rpcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:21:15.652794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.652674    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpcs\" (UniqueName: \"kubernetes.io/projected/e6120c16-264d-4a54-b4a3-7ba38464dbb7-kube-api-access-6rpcs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.652794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.652710    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6120c16-264d-4a54-b4a3-7ba38464dbb7-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:15.709583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.709547    4546 generic.go:356] "Generic (PLEG): container finished" podID="73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" containerID="a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5" exitCode=0
Nov 03 12:21:15.709740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.709589    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"
Nov 03 12:21:15.709740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.709635    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8" event={"ID":"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0","Type":"ContainerDied","Data":"a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5"}
Nov 03 12:21:15.709740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.709689    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8" event={"ID":"73c640c9-fcb6-4c15-bbbd-11787a1fa5f0","Type":"ContainerDied","Data":"ae43c2c42b81a515e55d1e4d1dc7169591e406d098cb78a56a904f241e856e80"}
Nov 03 12:21:15.709740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.709720    4546 scope.go:122] "RemoveContainer" containerID="a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5"
Nov 03 12:21:15.711547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.711475    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"
Nov 03 12:21:15.711665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.711481    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg" event={"ID":"3eb02115-8e78-48fa-97a6-b3282da727ca","Type":"ContainerDied","Data":"5127765d69b2bc02265a34453d99e2268020419dd2e6436d6c771e73bddde6ac"}
Nov 03 12:21:15.711665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.711607    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5127765d69b2bc02265a34453d99e2268020419dd2e6436d6c771e73bddde6ac"
Nov 03 12:21:15.713151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.713130    4546 generic.go:356] "Generic (PLEG): container finished" podID="e6120c16-264d-4a54-b4a3-7ba38464dbb7" containerID="b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a" exitCode=0
Nov 03 12:21:15.713266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.713172    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc" event={"ID":"e6120c16-264d-4a54-b4a3-7ba38464dbb7","Type":"ContainerDied","Data":"b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a"}
Nov 03 12:21:15.713266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.713195    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"
Nov 03 12:21:15.713266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.713212    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc" event={"ID":"e6120c16-264d-4a54-b4a3-7ba38464dbb7","Type":"ContainerDied","Data":"38c113649492ae35124a8fd5bb662d71a47e0057c8249ccd0cb8f076a1815743"}
Nov 03 12:21:15.723582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.723558    4546 scope.go:122] "RemoveContainer" containerID="a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5"
Nov 03 12:21:15.724262 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:21:15.724238    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5\": container with ID starting with a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5 not found: ID does not exist" containerID="a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5"
Nov 03 12:21:15.724376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.724273    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5"} err="failed to get container status \"a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5\": rpc error: code = NotFound desc = could not find container \"a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5\": container with ID starting with a820566f5831d4cf8365be543b76ad9fccd7dfffe91290317e09419259643cd5 not found: ID does not exist"
Nov 03 12:21:15.724376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.724299    4546 scope.go:122] "RemoveContainer" containerID="b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a"
Nov 03 12:21:15.735281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.735260    4546 scope.go:122] "RemoveContainer" containerID="b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a"
Nov 03 12:21:15.735628 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:21:15.735607    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a\": container with ID starting with b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a not found: ID does not exist" containerID="b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a"
Nov 03 12:21:15.735685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.735638    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a"} err="failed to get container status \"b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a\": rpc error: code = NotFound desc = could not find container \"b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a\": container with ID starting with b60919354d33681a91ab664fc94c972024313c3acf0f670c883e2b6e793e756a not found: ID does not exist"
Nov 03 12:21:15.921089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.920993    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"]
Nov 03 12:21:15.924108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:15.924076    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-56df5ff697-5crw8"]
Nov 03 12:21:16.018191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:16.017793    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"]
Nov 03 12:21:16.023055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:16.023024    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-84d5b5d5ff-vsbsc"]
Nov 03 12:21:16.575917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:16.575884    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" path="/var/lib/kubelet/pods/73c640c9-fcb6-4c15-bbbd-11787a1fa5f0/volumes"
Nov 03 12:21:16.576360 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:16.576346    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6120c16-264d-4a54-b4a3-7ba38464dbb7" path="/var/lib/kubelet/pods/e6120c16-264d-4a54-b4a3-7ba38464dbb7/volumes"
Nov 03 12:21:28.043829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:28.043795    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"]
Nov 03 12:21:28.090585 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:28.090557    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a037e0b-snr9r"]
Nov 03 12:21:28.575461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:28.575429    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b316da01-5d90-4e6d-b8eb-edfffae718c2" path="/var/lib/kubelet/pods/b316da01-5d90-4e6d-b8eb-edfffae718c2/volumes"
Nov 03 12:21:33.899459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.899422    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"]
Nov 03 12:21:33.900319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900297    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e6120c16-264d-4a54-b4a3-7ba38464dbb7" containerName="payload-pre-processing"
Nov 03 12:21:33.900429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900320    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6120c16-264d-4a54-b4a3-7ba38464dbb7" containerName="payload-pre-processing"
Nov 03 12:21:33.900429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900340    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3eb02115-8e78-48fa-97a6-b3282da727ca" containerName="revoke-keys"
Nov 03 12:21:33.900429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900347    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb02115-8e78-48fa-97a6-b3282da727ca" containerName="revoke-keys"
Nov 03 12:21:33.900429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900425    4546 cpu_manager.go:414] "containerMap: removing container" podUID="73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" containerName="payload-processing"
Nov 03 12:21:33.900639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900434    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" containerName="payload-processing"
Nov 03 12:21:33.900639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900574    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c640c9-fcb6-4c15-bbbd-11787a1fa5f0" containerName="payload-processing"
Nov 03 12:21:33.900639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900588    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6120c16-264d-4a54-b4a3-7ba38464dbb7" containerName="payload-pre-processing"
Nov 03 12:21:33.900639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.900599    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb02115-8e78-48fa-97a6-b3282da727ca" containerName="revoke-keys"
Nov 03 12:21:33.905464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.905443    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:33.955381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.955354    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-deb666\""
Nov 03 12:21:33.990246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.990212    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:33.990529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:33.990504    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjd2f\" (UniqueName: \"kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f\") pod \"payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.001553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.001526    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-deb666-dockercfg-zfjd8\""
Nov 03 12:21:34.005029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.005006    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"]
Nov 03 12:21:34.091882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.091847    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.092063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.092031    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjd2f\" (UniqueName: \"kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f\") pod \"payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.092550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.092530    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.145371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.145333    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjd2f\" (UniqueName: \"kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f\") pod \"payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.239192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.239159    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.356818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.356785    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"]
Nov 03 12:21:34.363768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.363733    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.396345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.396316    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.396514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.396354    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs92\" (UniqueName: \"kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92\") pod \"payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.444187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.444158    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"]
Nov 03 12:21:34.497400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.497329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.497400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.497366    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs92\" (UniqueName: \"kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92\") pod \"payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.498272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.498238    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.550731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.550700    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs92\" (UniqueName: \"kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92\") pod \"payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.638054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.635728    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"]
Nov 03 12:21:34.638302 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:34.638275    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91625b58_be1c_48af_a2d9_1b7d143e7e4c.slice/crio-e95892c46242de2e269daf5e548f64255e0fed8e89b65ddda529312c5d98a424 WatchSource:0}: Error finding container e95892c46242de2e269daf5e548f64255e0fed8e89b65ddda529312c5d98a424: Status 404 returned error can't find the container with id e95892c46242de2e269daf5e548f64255e0fed8e89b65ddda529312c5d98a424
Nov 03 12:21:34.678672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.678645    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:34.856059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.856020    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5" event={"ID":"91625b58-be1c-48af-a2d9-1b7d143e7e4c","Type":"ContainerStarted","Data":"d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6"}
Nov 03 12:21:34.856239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.856078    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5" event={"ID":"91625b58-be1c-48af-a2d9-1b7d143e7e4c","Type":"ContainerStarted","Data":"e95892c46242de2e269daf5e548f64255e0fed8e89b65ddda529312c5d98a424"}
Nov 03 12:21:34.856441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.856416    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:34.963172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:34.963110    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5" podStartSLOduration=1.963089155 podStartE2EDuration="1.963089155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:34.96188572 +0000 UTC m=+1836.533469633" watchObservedRunningTime="2026-11-03 12:21:34.963089155 +0000 UTC m=+1836.534673068"
Nov 03 12:21:35.098004 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:35.091019    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf014f23e_d756_4dbc_9dc7_65928b1cc013.slice/crio-bd340dc20015347d030d16fc4e8c60faf9f2a6abb2bace8f56806a81fb79c5d2 WatchSource:0}: Error finding container bd340dc20015347d030d16fc4e8c60faf9f2a6abb2bace8f56806a81fb79c5d2: Status 404 returned error can't find the container with id bd340dc20015347d030d16fc4e8c60faf9f2a6abb2bace8f56806a81fb79c5d2
Nov 03 12:21:35.098004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:35.091076    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"]
Nov 03 12:21:35.866406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:35.866373    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9" event={"ID":"f014f23e-d756-4dbc-9dc7-65928b1cc013","Type":"ContainerStarted","Data":"ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579"}
Nov 03 12:21:35.866612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:35.866434    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9" event={"ID":"f014f23e-d756-4dbc-9dc7-65928b1cc013","Type":"ContainerStarted","Data":"bd340dc20015347d030d16fc4e8c60faf9f2a6abb2bace8f56806a81fb79c5d2"}
Nov 03 12:21:35.866681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:35.866648    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:35.981372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:35.981312    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9" podStartSLOduration=1.9812932939999999 podStartE2EDuration="1.981293294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:35.976411538 +0000 UTC m=+1837.547995452" watchObservedRunningTime="2026-11-03 12:21:35.981293294 +0000 UTC m=+1837.552877209"
Nov 03 12:21:37.652381 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:21:37.652296    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:21:40.906778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:40.906729    4546 generic.go:356] "Generic (PLEG): container finished" podID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerID="7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10" exitCode=0
Nov 03 12:21:40.907286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:40.906813    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" event={"ID":"b2f3b83c-ac0b-47c3-93a5-441f9d349054","Type":"ContainerDied","Data":"7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"}
Nov 03 12:21:40.907286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:40.907239    4546 scope.go:122] "RemoveContainer" containerID="7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"
Nov 03 12:21:40.908712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:40.908684    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:21:40.908790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:40.908772    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:21:40.908832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:40.908808    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:21:41.915249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:41.915212    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" event={"ID":"b2f3b83c-ac0b-47c3-93a5-441f9d349054","Type":"ContainerStarted","Data":"2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca"}
Nov 03 12:21:42.177787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:42.177686    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:21:42.178652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:42.178632    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 12:21:42.178735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:42.178689    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 12:21:43.177271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:43.177236    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 12:21:43.177700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:43.177294    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 12:21:44.177807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.177776    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 12:21:44.178219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.177840    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 12:21:44.181093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.181068    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:21:44.736771 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.733237    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"]
Nov 03 12:21:44.737941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.737925    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:44.773111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.773081    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:44.773293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.773279    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7slg\" (UniqueName: \"kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg\") pod \"payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:44.783330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.783301    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"]
Nov 03 12:21:44.831825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.826100    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1e151c-dockercfg-gsw7c\""
Nov 03 12:21:44.831825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.826886    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1e151c\""
Nov 03 12:21:44.874217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.874191    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7slg\" (UniqueName: \"kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg\") pod \"payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:44.874388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.874312    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:44.875052 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.875024    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:44.933479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:44.933448    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7slg\" (UniqueName: \"kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg\") pod \"payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:45.051787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.051674    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:45.177531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.177183    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 12:21:45.177531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.177229    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 12:21:45.184837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.184811    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"]
Nov 03 12:21:45.191717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.191697    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.261717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.261675    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"]
Nov 03 12:21:45.281237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.281208    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gm9t\" (UniqueName: \"kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.281411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.281343    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.383429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.383346    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gm9t\" (UniqueName: \"kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.383587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.383473    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.384232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.384211    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.437782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.436916    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gm9t\" (UniqueName: \"kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t\") pod \"payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.456458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.456425    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"]
Nov 03 12:21:45.457231 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:45.457202    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23e16d8_2fa1_45bf_b110_02a0930ffd0f.slice/crio-9b95db4eaf8a4882ee8058242ec5b8a4c4804e0941458f328e926caf1d982e03 WatchSource:0}: Error finding container 9b95db4eaf8a4882ee8058242ec5b8a4c4804e0941458f328e926caf1d982e03: Status 404 returned error can't find the container with id 9b95db4eaf8a4882ee8058242ec5b8a4c4804e0941458f328e926caf1d982e03
Nov 03 12:21:45.535069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.535033    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:45.869209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.869181    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:21:45.922931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.921768    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"]
Nov 03 12:21:45.962241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.962168    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr" event={"ID":"3c3846e6-c03e-40e1-8551-34edbdd379b6","Type":"ContainerStarted","Data":"9e3c1fd1ed2b937d8c6654f5a22ad4ec41990ffd6b3dc6591e5326b3a31d2447"}
Nov 03 12:21:45.964046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.963982    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5" event={"ID":"e23e16d8-2fa1-45bf-b110-02a0930ffd0f","Type":"ContainerStarted","Data":"92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58"}
Nov 03 12:21:45.964046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.964022    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5" event={"ID":"e23e16d8-2fa1-45bf-b110-02a0930ffd0f","Type":"ContainerStarted","Data":"9b95db4eaf8a4882ee8058242ec5b8a4c4804e0941458f328e926caf1d982e03"}
Nov 03 12:21:45.964192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:45.964137    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:46.103790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:46.103718    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5" podStartSLOduration=2.103698729 podStartE2EDuration="2.103698729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:46.100964338 +0000 UTC m=+1847.672548252" watchObservedRunningTime="2026-11-03 12:21:46.103698729 +0000 UTC m=+1847.675282664"
Nov 03 12:21:46.177448 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:46.177367    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 12:21:46.177448 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:46.177424    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 12:21:46.875650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:46.875618    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:21:46.972325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:46.972285    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr" event={"ID":"3c3846e6-c03e-40e1-8551-34edbdd379b6","Type":"ContainerStarted","Data":"f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21"}
Nov 03 12:21:47.098982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:47.098923    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr" podStartSLOduration=2.098903492 podStartE2EDuration="2.098903492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:47.097000619 +0000 UTC m=+1848.668584577" watchObservedRunningTime="2026-11-03 12:21:47.098903492 +0000 UTC m=+1848.670487405"
Nov 03 12:21:47.177866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:47.177782    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 12:21:47.177866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:47.177838    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 12:21:47.979203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:47.979174    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:48.182462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:48.182429    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:21:48.183594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:48.183571    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:21:50.067394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.067362    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"]
Nov 03 12:21:50.071932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.071909    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.124863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.122510    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-059838-openshift-default-dockercfg-gz94q\""
Nov 03 12:21:50.172766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.172722    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"]
Nov 03 12:21:50.182416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182368    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.182588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182449    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqt8\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.182588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182505    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.182724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182623    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.182848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182734    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.182848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182802    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.182848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182838    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.183032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182854    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.183032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.182909    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.285027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.284947    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.285866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.285841    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.286040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.286025    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.286188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.286172    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.286375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.286359    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.286540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.286523    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.286742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.286724    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.286918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.286901    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqt8\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.287079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.287060    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.287976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.287937    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.296832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.288363    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.296832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.288593    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.297022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.289176    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.297022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.289535    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.297022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.293385    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.316692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.316659    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.342509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.342434    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.343416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.343397    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqt8\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8\") pod \"e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") " pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.386250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.386220    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:50.804270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.804241    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"]
Nov 03 12:21:50.805041 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:50.805019    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867be5c5_4007_4c7b_b4f2_b89440379906.slice/crio-ba07cefca6b58b90c4ca9f6334072f47b276e8ca47ff5fe6eba4851594cac35c WatchSource:0}: Error finding container ba07cefca6b58b90c4ca9f6334072f47b276e8ca47ff5fe6eba4851594cac35c: Status 404 returned error can't find the container with id ba07cefca6b58b90c4ca9f6334072f47b276e8ca47ff5fe6eba4851594cac35c
Nov 03 12:21:50.807683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.807649    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:21:50.807863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.807727    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:21:50.807863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:50.807794    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:21:51.001020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:51.000987    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" event={"ID":"867be5c5-4007-4c7b-b4f2-b89440379906","Type":"ContainerStarted","Data":"aa284368b47faaeced64cf6cc6d1b1346da47749e86d88fb80be801d3bd65f01"}
Nov 03 12:21:51.001187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:51.001041    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" event={"ID":"867be5c5-4007-4c7b-b4f2-b89440379906","Type":"ContainerStarted","Data":"ba07cefca6b58b90c4ca9f6334072f47b276e8ca47ff5fe6eba4851594cac35c"}
Nov 03 12:21:51.114102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:51.114052    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podStartSLOduration=1.11403774 podStartE2EDuration="1.11403774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:51.110787479 +0000 UTC m=+1852.682371390" watchObservedRunningTime="2026-11-03 12:21:51.11403774 +0000 UTC m=+1852.685621651"
Nov 03 12:21:51.387371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:51.387288    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:51.393360 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:51.393334    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:52.008418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:52.008389    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:52.009457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:52.009436    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:21:52.779708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:52.779669    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"]
Nov 03 12:21:52.780262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:52.779956    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy" containerID="cri-o://466fc8f0c46162286c58091b7531689b0289e638cb217761b186bf3884d9f683" gracePeriod=30
Nov 03 12:21:56.975428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:56.975401    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:21:57.715253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.715221    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"]
Nov 03 12:21:57.719699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.719677    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:57.769299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.769266    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-059838\""
Nov 03 12:21:57.769560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.769544    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-059838-dockercfg-56rnz\""
Nov 03 12:21:57.774464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.774405    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"]
Nov 03 12:21:57.801784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.796976    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:57.801784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.797084    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4244w\" (UniqueName: \"kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w\") pod \"payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:57.920362 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.907524    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"]
Nov 03 12:21:57.920362 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.918978    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:57.927572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.927537    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:57.928048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.927978    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hc8\" (UniqueName: \"kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8\") pod \"payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:57.928369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.928349    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:57.931241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.931219    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4244w\" (UniqueName: \"kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w\") pod \"payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:57.940374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.931162    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:57.954453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.954425    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"]
Nov 03 12:21:57.993844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:57.993681    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4244w\" (UniqueName: \"kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w\") pod \"payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:58.037909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.037861    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hc8\" (UniqueName: \"kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8\") pod \"payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:58.038198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.038179    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:58.039221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.039191    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:58.040230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.040208    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:58.054596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.054572    4546 generic.go:356] "Generic (PLEG): container finished" podID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerID="466fc8f0c46162286c58091b7531689b0289e638cb217761b186bf3884d9f683" exitCode=0
Nov 03 12:21:58.054786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.054607    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" event={"ID":"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f","Type":"ContainerDied","Data":"466fc8f0c46162286c58091b7531689b0289e638cb217761b186bf3884d9f683"}
Nov 03 12:21:58.095226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.095191    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hc8\" (UniqueName: \"kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8\") pod \"payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:58.152814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.151921    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:21:58.252640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.252558    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343423    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343512    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343623    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343672    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343783    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343876    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.343942    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.344031    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.344074    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket\") pod \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\" (UID: \"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f\") "
Nov 03 12:21:58.345918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.345477    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:21:58.345918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.345614    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:21:58.347838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.347809    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:21:58.348274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.348210    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:21:58.348477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.348435    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:21:58.348681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.348653    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:21:58.351398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.351361    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:21:58.353494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.353460    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "kube-api-access-7t8z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:21:58.357070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.357046    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo" pod "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" (UID: "0ec9fec2-2b54-42f0-bbb8-f85f8233b84f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:21:58.443366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.443336    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"]
Nov 03 12:21:58.444654 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:58.444629    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e384abe_3c71_414e_adf5_c4668a6c7b98.slice/crio-48fcc3606ed2c91433dfbeeaa06ce50ad2e6681b7c56a24cfbd2b6f48a78d1cf WatchSource:0}: Error finding container 48fcc3606ed2c91433dfbeeaa06ce50ad2e6681b7c56a24cfbd2b6f48a78d1cf: Status 404 returned error can't find the container with id 48fcc3606ed2c91433dfbeeaa06ce50ad2e6681b7c56a24cfbd2b6f48a78d1cf
Nov 03 12:21:58.446723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446693    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446717    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446731    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446765    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446780    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446794    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446808    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446822    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t8z6\" (UniqueName: \"kubernetes.io/projected/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-kube-api-access-7t8z6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.446860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.446836    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:21:58.635295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.635266    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"]
Nov 03 12:21:58.638654 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:21:58.638625    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc052d6b6_c238_4cb4_9bd9_b839179ef2b2.slice/crio-bf52839711a5eb926798d62cabfe39e4268e0a4ce347962e4fbfc6bdfff7aa1a WatchSource:0}: Error finding container bf52839711a5eb926798d62cabfe39e4268e0a4ce347962e4fbfc6bdfff7aa1a: Status 404 returned error can't find the container with id bf52839711a5eb926798d62cabfe39e4268e0a4ce347962e4fbfc6bdfff7aa1a
Nov 03 12:21:58.987959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:58.987924    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:21:59.068930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.068882    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr" event={"ID":"c052d6b6-c238-4cb4-9bd9-b839179ef2b2","Type":"ContainerStarted","Data":"e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e"}
Nov 03 12:21:59.069347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.068949    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr" event={"ID":"c052d6b6-c238-4cb4-9bd9-b839179ef2b2","Type":"ContainerStarted","Data":"bf52839711a5eb926798d62cabfe39e4268e0a4ce347962e4fbfc6bdfff7aa1a"}
Nov 03 12:21:59.069347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.068971    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:21:59.070580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.070557    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"
Nov 03 12:21:59.070719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.070577    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t" event={"ID":"0ec9fec2-2b54-42f0-bbb8-f85f8233b84f","Type":"ContainerDied","Data":"8d1a66f5b1825760c3e4bbdcfeee3c1a55cb158d9fbffa38b35ce73a2d94504e"}
Nov 03 12:21:59.070719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.070631    4546 scope.go:122] "RemoveContainer" containerID="466fc8f0c46162286c58091b7531689b0289e638cb217761b186bf3884d9f683"
Nov 03 12:21:59.072223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.072200    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l" event={"ID":"3e384abe-3c71-414e-adf5-c4668a6c7b98","Type":"ContainerStarted","Data":"8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f"}
Nov 03 12:21:59.072694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.072677    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:21:59.072946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.072923    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l" event={"ID":"3e384abe-3c71-414e-adf5-c4668a6c7b98","Type":"ContainerStarted","Data":"48fcc3606ed2c91433dfbeeaa06ce50ad2e6681b7c56a24cfbd2b6f48a78d1cf"}
Nov 03 12:21:59.197515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.197431    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"]
Nov 03 12:21:59.206162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.206129    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-c464t"]
Nov 03 12:21:59.304935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.304876    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l" podStartSLOduration=2.304862377 podStartE2EDuration="2.304862377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:59.302384711 +0000 UTC m=+1860.873968622" watchObservedRunningTime="2026-11-03 12:21:59.304862377 +0000 UTC m=+1860.876446289"
Nov 03 12:21:59.407388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:21:59.407331    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr" podStartSLOduration=2.407314161 podStartE2EDuration="2.407314161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:21:59.406430005 +0000 UTC m=+1860.978013918" watchObservedRunningTime="2026-11-03 12:21:59.407314161 +0000 UTC m=+1860.978898094"
Nov 03 12:22:00.574374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:00.574341    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" path="/var/lib/kubelet/pods/0ec9fec2-2b54-42f0-bbb8-f85f8233b84f/volumes"
Nov 03 12:22:06.755797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.755742    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"]
Nov 03 12:22:06.756650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.756627    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy"
Nov 03 12:22:06.756783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.756653    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy"
Nov 03 12:22:06.757038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.757020    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec9fec2-2b54-42f0-bbb8-f85f8233b84f" containerName="istio-proxy"
Nov 03 12:22:06.762186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.762166    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:06.838389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.838336    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"]
Nov 03 12:22:06.861543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.861442    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45b1b7\""
Nov 03 12:22:06.861543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.861502    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45b1b7-dockercfg-fnnf8\""
Nov 03 12:22:06.895606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.895558    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:06.895805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.895780    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh\") pod \"payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:06.997337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.997297    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh\") pod \"payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:06.997616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.997583    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:06.998381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:06.998360    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:07.051969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.051895    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh\") pod \"payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:07.076153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.076121    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:07.248669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.245281    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"]
Nov 03 12:22:07.266656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.266626    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.283295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.283220    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"]
Nov 03 12:22:07.403466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.403373    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.403466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.403409    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c246d\" (UniqueName: \"kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.485089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.485055    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"]
Nov 03 12:22:07.485422 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:22:07.485401    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5d16af_69e6_4486_a116_66891547be99.slice/crio-400ab58aa31ea1876d19fe537342d670d7ac814c7023e9596741e1e2a3db263d WatchSource:0}: Error finding container 400ab58aa31ea1876d19fe537342d670d7ac814c7023e9596741e1e2a3db263d: Status 404 returned error can't find the container with id 400ab58aa31ea1876d19fe537342d670d7ac814c7023e9596741e1e2a3db263d
Nov 03 12:22:07.504679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.504648    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.504802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.504691    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c246d\" (UniqueName: \"kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.505336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.505314    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.557495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.557465    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c246d\" (UniqueName: \"kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d\") pod \"payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.587661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.587638    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:07.667698 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:07.667618    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:22:07.997202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:07.997166    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"]
Nov 03 12:22:07.998720 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:22:07.998697    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode793873c_c9da_437b_bfa3_06a9be0ef4f8.slice/crio-710638a0f5b138e003e4e7cc5fc5bf2e0f0ef5fe29abe92d109aac0b96c1ebb5 WatchSource:0}: Error finding container 710638a0f5b138e003e4e7cc5fc5bf2e0f0ef5fe29abe92d109aac0b96c1ebb5: Status 404 returned error can't find the container with id 710638a0f5b138e003e4e7cc5fc5bf2e0f0ef5fe29abe92d109aac0b96c1ebb5
Nov 03 12:22:08.169789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.167776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr" event={"ID":"e793873c-c9da-437b-bfa3-06a9be0ef4f8","Type":"ContainerStarted","Data":"85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd"}
Nov 03 12:22:08.169789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.167838    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr" event={"ID":"e793873c-c9da-437b-bfa3-06a9be0ef4f8","Type":"ContainerStarted","Data":"710638a0f5b138e003e4e7cc5fc5bf2e0f0ef5fe29abe92d109aac0b96c1ebb5"}
Nov 03 12:22:08.169789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.168880    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:08.171360 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.171328    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5" event={"ID":"ac5d16af-69e6-4486-a116-66891547be99","Type":"ContainerStarted","Data":"cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693"}
Nov 03 12:22:08.171506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.171378    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5" event={"ID":"ac5d16af-69e6-4486-a116-66891547be99","Type":"ContainerStarted","Data":"400ab58aa31ea1876d19fe537342d670d7ac814c7023e9596741e1e2a3db263d"}
Nov 03 12:22:08.171998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.171977    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:08.289608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.289494    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr" podStartSLOduration=1.289478674 podStartE2EDuration="1.289478674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:22:08.28705155 +0000 UTC m=+1869.858635486" watchObservedRunningTime="2026-11-03 12:22:08.289478674 +0000 UTC m=+1869.861062586"
Nov 03 12:22:08.396581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:08.396525    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5" podStartSLOduration=2.3965084 podStartE2EDuration="2.3965084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:22:08.393527033 +0000 UTC m=+1869.965110952" watchObservedRunningTime="2026-11-03 12:22:08.3965084 +0000 UTC m=+1869.968092313"
Nov 03 12:22:09.046366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:09.046333    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"]
Nov 03 12:22:09.049838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:09.049808    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a7ffe0-mhvfc"]
Nov 03 12:22:09.091631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:09.091593    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"]
Nov 03 12:22:09.094853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:09.094834    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17fc161d-q9rtt"]
Nov 03 12:22:10.082413 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:10.082387    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:22:10.575199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:10.575164    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4a523a-0a1e-42d8-9c19-5561579e7d95" path="/var/lib/kubelet/pods/4b4a523a-0a1e-42d8-9c19-5561579e7d95/volumes"
Nov 03 12:22:10.575639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:10.575625    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84514f25-e4c6-477a-93b9-662eaa4ff408" path="/var/lib/kubelet/pods/84514f25-e4c6-477a-93b9-662eaa4ff408/volumes"
Nov 03 12:22:11.089741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:11.089709    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:22:13.583926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:13.583882    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:22:13.584393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:13.583968    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:22:16.229069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:16.229041    4546 generic.go:356] "Generic (PLEG): container finished" podID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerID="f36b9438b77c59ac0c3097abec447996adc816795a2c72e57169c4cf656d460c" exitCode=0
Nov 03 12:22:16.229526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:16.229116    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" event={"ID":"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f","Type":"ContainerDied","Data":"f36b9438b77c59ac0c3097abec447996adc816795a2c72e57169c4cf656d460c"}
Nov 03 12:22:16.229526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:16.229483    4546 scope.go:122] "RemoveContainer" containerID="f36b9438b77c59ac0c3097abec447996adc816795a2c72e57169c4cf656d460c"
Nov 03 12:22:16.230924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:16.230893    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:22:16.231042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:16.230982    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:22:16.231042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:16.231032    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:22:17.240461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:17.240430    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" event={"ID":"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f","Type":"ContainerStarted","Data":"802b3dedab59b0e7deed1c840878c01d36c8c35de62be2ed12a79c9d53db2a37"}
Nov 03 12:22:17.757094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:17.757059    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:22:17.757293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:17.757276    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 12:22:17.757346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:17.757332    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 12:22:18.757135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:18.757101    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 12:22:18.757553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:18.757156    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 12:22:18.949743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:18.949703    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"]
Nov 03 12:22:19.049987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:19.049894    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"]
Nov 03 12:22:19.756982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:19.756946    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 12:22:19.757170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:19.756996    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 12:22:20.186366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:20.186290    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:22:20.186519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:20.186367    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:22:20.757146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:20.757110    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 12:22:20.757334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:20.757168    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 12:22:21.757188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:21.757150    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 12:22:21.757389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:21.757250    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 12:22:22.761028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:22.760998    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:22:22.761562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:22.761544    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:22:22.762405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:22.762390    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:22:23.442589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.442558    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"]
Nov 03 12:22:23.450997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.448191    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69fd8043-xngrg"]
Nov 03 12:22:23.695271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.695179    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"]
Nov 03 12:22:23.695473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.695443    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy" containerID="cri-o://f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784" gracePeriod=30
Nov 03 12:22:23.701658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.701628    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"]
Nov 03 12:22:23.701952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.701925    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy" containerID="cri-o://2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79" gracePeriod=30
Nov 03 12:22:23.707277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.707248    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"]
Nov 03 12:22:23.707533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.707498    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy" containerID="cri-o://316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38" gracePeriod=30
Nov 03 12:22:23.717518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.717487    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"]
Nov 03 12:22:23.717776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.717731    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy" containerID="cri-o://8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e" gracePeriod=30
Nov 03 12:22:23.950214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:23.950078    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq" podUID="f81a81b7-2c19-423d-8a17-fa7e44c926db" containerName="payload-processing" containerID="cri-o://320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56" gracePeriod=25
Nov 03 12:22:24.051079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.051038    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j" podUID="1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" containerName="payload-pre-processing" containerID="cri-o://2394fd69dde59829933fcffd25e57be8037208725ad4edb78734e2efdab0f5bf" gracePeriod=25
Nov 03 12:22:24.247095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.247065    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:22:24.303459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.303426    4546 generic.go:356] "Generic (PLEG): container finished" podID="1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" containerID="2394fd69dde59829933fcffd25e57be8037208725ad4edb78734e2efdab0f5bf" exitCode=0
Nov 03 12:22:24.303641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.303497    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j" event={"ID":"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48","Type":"ContainerDied","Data":"2394fd69dde59829933fcffd25e57be8037208725ad4edb78734e2efdab0f5bf"}
Nov 03 12:22:24.305162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.305128    4546 generic.go:356] "Generic (PLEG): container finished" podID="f81a81b7-2c19-423d-8a17-fa7e44c926db" containerID="320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56" exitCode=0
Nov 03 12:22:24.305311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.305158    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq" event={"ID":"f81a81b7-2c19-423d-8a17-fa7e44c926db","Type":"ContainerDied","Data":"320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56"}
Nov 03 12:22:24.305311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.305236    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq" event={"ID":"f81a81b7-2c19-423d-8a17-fa7e44c926db","Type":"ContainerDied","Data":"4a1319234c46bcbe84a432d33f5c42a767e1d6d45cdf3f01266005fa62cc5bc8"}
Nov 03 12:22:24.305311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.305273    4546 scope.go:122] "RemoveContainer" containerID="320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56"
Nov 03 12:22:24.305537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.305521    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"
Nov 03 12:22:24.318304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.318283    4546 scope.go:122] "RemoveContainer" containerID="320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56"
Nov 03 12:22:24.318577 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:24.318555    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56\": container with ID starting with 320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56 not found: ID does not exist" containerID="320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56"
Nov 03 12:22:24.318641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.318590    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56"} err="failed to get container status \"320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56\": rpc error: code = NotFound desc = could not find container \"320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56\": container with ID starting with 320c39e63547ba15b2644eb09d33a0186d4d62bbf895721915a36c8e43972d56 not found: ID does not exist"
Nov 03 12:22:24.329075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.329053    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume\") pod \"f81a81b7-2c19-423d-8a17-fa7e44c926db\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") "
Nov 03 12:22:24.329211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.329165    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg\" (UniqueName: \"kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg\") pod \"f81a81b7-2c19-423d-8a17-fa7e44c926db\" (UID: \"f81a81b7-2c19-423d-8a17-fa7e44c926db\") "
Nov 03 12:22:24.329434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.329412    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume" pod "f81a81b7-2c19-423d-8a17-fa7e44c926db" (UID: "f81a81b7-2c19-423d-8a17-fa7e44c926db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:24.329919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.329815    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81a81b7-2c19-423d-8a17-fa7e44c926db-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:24.331423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.331396    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg" pod "f81a81b7-2c19-423d-8a17-fa7e44c926db" (UID: "f81a81b7-2c19-423d-8a17-fa7e44c926db"). InnerVolumeSpecName "kube-api-access-4kqtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:24.356405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.356380    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:22:24.430788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.430741    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume\") pod \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") "
Nov 03 12:22:24.430939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.430851    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7\" (UniqueName: \"kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7\") pod \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\" (UID: \"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48\") "
Nov 03 12:22:24.431044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.431024    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume" pod "1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" (UID: "1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:24.431291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.431276    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kqtg\" (UniqueName: \"kubernetes.io/projected/f81a81b7-2c19-423d-8a17-fa7e44c926db-kube-api-access-4kqtg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:24.431343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.431293    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:24.433069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.433050    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7" pod "1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" (UID: "1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48"). InnerVolumeSpecName "kube-api-access-x29s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:24.532230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.532195    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29s7\" (UniqueName: \"kubernetes.io/projected/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48-kube-api-access-x29s7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:24.575019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.574988    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb02115-8e78-48fa-97a6-b3282da727ca" path="/var/lib/kubelet/pods/3eb02115-8e78-48fa-97a6-b3282da727ca/volumes"
Nov 03 12:22:24.712632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.712604    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"]
Nov 03 12:22:24.718796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:24.718772    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-5b747995b5-2frdq"]
Nov 03 12:22:25.314598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:25.314522    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"
Nov 03 12:22:25.315066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:25.314522    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j" event={"ID":"1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48","Type":"ContainerDied","Data":"fb103340eef76110a63e7d6c03ef5da0db5e2c85b09ead6191d8f3232bd554ce"}
Nov 03 12:22:25.315066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:25.314677    4546 scope.go:122] "RemoveContainer" containerID="2394fd69dde59829933fcffd25e57be8037208725ad4edb78734e2efdab0f5bf"
Nov 03 12:22:25.467798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:25.467767    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"]
Nov 03 12:22:25.471130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:25.471107    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-7bd7f9465f-twt9j"]
Nov 03 12:22:26.574275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:26.574242    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" path="/var/lib/kubelet/pods/1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48/volumes"
Nov 03 12:22:26.574647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:26.574640    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81a81b7-2c19-423d-8a17-fa7e44c926db" path="/var/lib/kubelet/pods/f81a81b7-2c19-423d-8a17-fa7e44c926db/volumes"
Nov 03 12:22:29.028672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.028639    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:22:29.091397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091372    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091419    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091449    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091516    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091551    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091582    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091644    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091775    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.091950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091822    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25\") pod \"91fd6e99-9ce1-4b15-a369-48f35701384d\" (UID: \"91fd6e99-9ce1-4b15-a369-48f35701384d\") "
Nov 03 12:22:29.092064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.091947    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.092123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092083    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.092181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092159    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.092292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092272    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.092548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092529    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.092649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092549    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.092649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092548    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.092649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092559    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.092649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.092569    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.095073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.095035    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.095425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.095394    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.096240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.095607    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.097882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.097860    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25" pod "91fd6e99-9ce1-4b15-a369-48f35701384d" (UID: "91fd6e99-9ce1-4b15-a369-48f35701384d"). InnerVolumeSpecName "kube-api-access-78q25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.100965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.100944    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:22:29.109083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.109060    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:22:29.113179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.113163    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:22:29.193796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193727    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.193796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193808    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193842    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193901    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193928    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193969    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.193993    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194025    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.194060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194058    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194089    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194119    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194131    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194219    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194254    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194284    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194328    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194334    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194383    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194396    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194425    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194447    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194542    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194609    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194639    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194723    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194823    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.194929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194898    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194938    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194927    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194966    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.194994    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.195068    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token\") pod \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\" (UID: \"adcb9967-a55d-4625-9b2f-c9af04c7ef2a\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.195125    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.195169    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.195233    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data\") pod \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\" (UID: \"9760034f-aaab-4f63-a7f6-c5740e1ff5b0\") "
Nov 03 12:22:29.195404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.195281    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert\") pod \"431cccbb-c8e4-4b47-9e87-97819e146a4a\" (UID: \"431cccbb-c8e4-4b47-9e87-97819e146a4a\") "
Nov 03 12:22:29.196082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196063    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78q25\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-kube-api-access-78q25\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196080    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196089    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196100    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196108    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196117    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196125    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196133    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196142    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196150    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196160    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196168    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd6e99-9ce1-4b15-a369-48f35701384d-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196175    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196183    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd6e99-9ce1-4b15-a369-48f35701384d-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.196827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.196451    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.198110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.197504    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.198110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.197556    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.198110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.197691    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.198645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.198614    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.198645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.198627    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.198941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.198834    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.198941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.198856    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.200333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.200306    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.201044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.201022    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.201453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.201428    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.201641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.201585    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.201641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.201652    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "kube-api-access-tl5nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.201641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.201685    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.202008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.201786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "kube-api-access-52lfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.202643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.202615    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token" pod "431cccbb-c8e4-4b47-9e87-97819e146a4a" (UID: "431cccbb-c8e4-4b47-9e87-97819e146a4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.203178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.203160    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token" pod "adcb9967-a55d-4625-9b2f-c9af04c7ef2a" (UID: "adcb9967-a55d-4625-9b2f-c9af04c7ef2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.203241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.203192    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7" pod "9760034f-aaab-4f63-a7f6-c5740e1ff5b0" (UID: "9760034f-aaab-4f63-a7f6-c5740e1ff5b0"). InnerVolumeSpecName "kube-api-access-lrzj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.297688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297593    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.297688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297630    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5nf\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-kube-api-access-tl5nf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.297688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297647    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.297688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297660    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrzj7\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-kube-api-access-lrzj7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.297688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297676    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52lfn\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-kube-api-access-52lfn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.297688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297691    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297704    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297718    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297732    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297762    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297778    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297790    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297803    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297816    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431cccbb-c8e4-4b47-9e87-97819e146a4a-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297829    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431cccbb-c8e4-4b47-9e87-97819e146a4a-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297844    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297856    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760034f-aaab-4f63-a7f6-c5740e1ff5b0-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.298048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.297869    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb9967-a55d-4625-9b2f-c9af04c7ef2a-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.344437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.344407    4546 generic.go:356] "Generic (PLEG): container finished" podID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerID="f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784" exitCode=0
Nov 03 12:22:29.344638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.344477    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"
Nov 03 12:22:29.344638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.344497    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" event={"ID":"91fd6e99-9ce1-4b15-a369-48f35701384d","Type":"ContainerDied","Data":"f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784"}
Nov 03 12:22:29.344638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.344564    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j" event={"ID":"91fd6e99-9ce1-4b15-a369-48f35701384d","Type":"ContainerDied","Data":"0ae2510103ee3f256ba6481788736190d9f8ea00506e44db297a5545d45e6bd2"}
Nov 03 12:22:29.344638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.344587    4546 scope.go:122] "RemoveContainer" containerID="f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784"
Nov 03 12:22:29.346269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.346244    4546 generic.go:356] "Generic (PLEG): container finished" podID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerID="8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e" exitCode=0
Nov 03 12:22:29.346380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.346354    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" event={"ID":"adcb9967-a55d-4625-9b2f-c9af04c7ef2a","Type":"ContainerDied","Data":"8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e"}
Nov 03 12:22:29.346504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.346489    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"
Nov 03 12:22:29.346584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.346509    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8" event={"ID":"adcb9967-a55d-4625-9b2f-c9af04c7ef2a","Type":"ContainerDied","Data":"8235dd3b7d31840bad4555d7b9f49cb741ae3bd6c6394b2d577bb08f5c40d98b"}
Nov 03 12:22:29.347986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.347966    4546 generic.go:356] "Generic (PLEG): container finished" podID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerID="2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79" exitCode=0
Nov 03 12:22:29.348103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.348019    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"
Nov 03 12:22:29.348177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.348112    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" event={"ID":"431cccbb-c8e4-4b47-9e87-97819e146a4a","Type":"ContainerDied","Data":"2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79"}
Nov 03 12:22:29.348177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.348153    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7" event={"ID":"431cccbb-c8e4-4b47-9e87-97819e146a4a","Type":"ContainerDied","Data":"0bd38c875de6665762ed0a14e4bdf0b2e236142d8f6ee5a4d4d2190b6d0df4ee"}
Nov 03 12:22:29.349644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.349548    4546 generic.go:356] "Generic (PLEG): container finished" podID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerID="316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38" exitCode=0
Nov 03 12:22:29.349644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.349574    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" event={"ID":"9760034f-aaab-4f63-a7f6-c5740e1ff5b0","Type":"ContainerDied","Data":"316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38"}
Nov 03 12:22:29.349644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.349588    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"
Nov 03 12:22:29.349644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.349606    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd" event={"ID":"9760034f-aaab-4f63-a7f6-c5740e1ff5b0","Type":"ContainerDied","Data":"8fbc52395ebb3905b4d77388625a63385b6ee8e9ed0b17f3f0adf6c2062b5050"}
Nov 03 12:22:29.361679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.361661    4546 scope.go:122] "RemoveContainer" containerID="a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"
Nov 03 12:22:29.375425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.375400    4546 scope.go:122] "RemoveContainer" containerID="f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784"
Nov 03 12:22:29.375721 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:29.375700    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784\": container with ID starting with f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784 not found: ID does not exist" containerID="f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784"
Nov 03 12:22:29.375830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.375732    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784"} err="failed to get container status \"f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784\": rpc error: code = NotFound desc = could not find container \"f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784\": container with ID starting with f7e9b6727b984af1ab26ab17dd5f8af334074ceb7b4ed0a16ef8631dcc66a784 not found: ID does not exist"
Nov 03 12:22:29.375830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.375774    4546 scope.go:122] "RemoveContainer" containerID="a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"
Nov 03 12:22:29.376104 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:29.376081    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3\": container with ID starting with a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3 not found: ID does not exist" containerID="a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"
Nov 03 12:22:29.376394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.376111    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3"} err="failed to get container status \"a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3\": rpc error: code = NotFound desc = could not find container \"a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3\": container with ID starting with a0431cfe2e947f162c159d9e943cdde4c8a78016ed521f81dddc272f31f7e2a3 not found: ID does not exist"
Nov 03 12:22:29.376394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.376132    4546 scope.go:122] "RemoveContainer" containerID="8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e"
Nov 03 12:22:29.386016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.385996    4546 scope.go:122] "RemoveContainer" containerID="8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e"
Nov 03 12:22:29.386266 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:29.386245    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e\": container with ID starting with 8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e not found: ID does not exist" containerID="8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e"
Nov 03 12:22:29.386363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.386274    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e"} err="failed to get container status \"8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e\": rpc error: code = NotFound desc = could not find container \"8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e\": container with ID starting with 8f69a8629528983b2fde53bf86a2017f6a73b309cd65b81394f83bcccc76316e not found: ID does not exist"
Nov 03 12:22:29.386363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.386289    4546 scope.go:122] "RemoveContainer" containerID="2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79"
Nov 03 12:22:29.395836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.395813    4546 scope.go:122] "RemoveContainer" containerID="2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79"
Nov 03 12:22:29.396133 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:29.396114    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79\": container with ID starting with 2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79 not found: ID does not exist" containerID="2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79"
Nov 03 12:22:29.396194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.396145    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79"} err="failed to get container status \"2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79\": rpc error: code = NotFound desc = could not find container \"2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79\": container with ID starting with 2c7a4dfe057d9d407d4d8be280623d335b40527e54b43acd5315a3c75d6f1d79 not found: ID does not exist"
Nov 03 12:22:29.396194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.396161    4546 scope.go:122] "RemoveContainer" containerID="316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38"
Nov 03 12:22:29.410278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.410257    4546 scope.go:122] "RemoveContainer" containerID="316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38"
Nov 03 12:22:29.411097 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:29.410728    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38\": container with ID starting with 316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38 not found: ID does not exist" containerID="316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38"
Nov 03 12:22:29.411254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.411219    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38"} err="failed to get container status \"316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38\": rpc error: code = NotFound desc = could not find container \"316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38\": container with ID starting with 316a2cd09622607affe351c66501652900e53145d954601a85e0e8c3028f9d38 not found: ID does not exist"
Nov 03 12:22:29.618331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.618242    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"]
Nov 03 12:22:29.622069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.622044    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-4w5bd"]
Nov 03 12:22:29.721861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.721820    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"]
Nov 03 12:22:29.726275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.726209    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-cr69j"]
Nov 03 12:22:29.827169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.827112    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"]
Nov 03 12:22:29.830820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.830797    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-jrvx8"]
Nov 03 12:22:29.932452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.932364    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"]
Nov 03 12:22:29.935901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:29.935871    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-wnxh7"]
Nov 03 12:22:30.574374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:30.574329    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" path="/var/lib/kubelet/pods/431cccbb-c8e4-4b47-9e87-97819e146a4a/volumes"
Nov 03 12:22:30.574897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:30.574882    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" path="/var/lib/kubelet/pods/91fd6e99-9ce1-4b15-a369-48f35701384d/volumes"
Nov 03 12:22:30.575361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:30.575348    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" path="/var/lib/kubelet/pods/9760034f-aaab-4f63-a7f6-c5740e1ff5b0/volumes"
Nov 03 12:22:30.576156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:30.576139    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" path="/var/lib/kubelet/pods/adcb9967-a55d-4625-9b2f-c9af04c7ef2a/volumes"
Nov 03 12:22:37.658353 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:37.658277    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:22:37.916301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.916198    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"]
Nov 03 12:22:37.917284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917255    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" containerName="payload-pre-processing"
Nov 03 12:22:37.917284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917283    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" containerName="payload-pre-processing"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917321    4546 cpu_manager.go:414] "containerMap: removing container" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917331    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917344    4546 cpu_manager.go:414] "containerMap: removing container" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917352    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917372    4546 cpu_manager.go:414] "containerMap: removing container" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917380    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917390    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f81a81b7-2c19-423d-8a17-fa7e44c926db" containerName="payload-processing"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917397    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81a81b7-2c19-423d-8a17-fa7e44c926db" containerName="payload-processing"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917412    4546 cpu_manager.go:414] "containerMap: removing container" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917419    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917465    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy"
Nov 03 12:22:37.917504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917475    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917630    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcb9967-a55d-4625-9b2f-c9af04c7ef2a" containerName="istio-proxy"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917653    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9760034f-aaab-4f63-a7f6-c5740e1ff5b0" containerName="istio-proxy"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917667    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1fe0d7-3764-42a6-bc17-0d8d18bd9b48" containerName="payload-pre-processing"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917682    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917692    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fd6e99-9ce1-4b15-a369-48f35701384d" containerName="istio-proxy"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917701    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="431cccbb-c8e4-4b47-9e87-97819e146a4a" containerName="istio-proxy"
Nov 03 12:22:37.918196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.917711    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81a81b7-2c19-423d-8a17-fa7e44c926db" containerName="payload-processing"
Nov 03 12:22:37.923948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.923915    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:37.972958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:37.972922    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"]
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.014352    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjksm\" (UniqueName: \"kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.014435    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.014696    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.014707    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.015080    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.015154    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:22:38.024330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.015182    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:22:38.116905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.116806    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.117099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.117005    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjksm\" (UniqueName: \"kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.117099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.117053    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.117764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.117724    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.120065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.120040    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.175822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.174937    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjksm\" (UniqueName: \"kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm\") pod \"maas-api-revoke-sub-8226ca55-lkknn\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.241776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.241122    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:38.991838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:38.991370    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"]
Nov 03 12:22:38.994189 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:22:38.994156    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5fff96_cfca_4d24_8b0e_9b4091aa994f.slice/crio-3a29dff9ab02074ed49d3fe1a5732ac06bbfcce88e1f1e7ed1cde6b4a6e2233a WatchSource:0}: Error finding container 3a29dff9ab02074ed49d3fe1a5732ac06bbfcce88e1f1e7ed1cde6b4a6e2233a: Status 404 returned error can't find the container with id 3a29dff9ab02074ed49d3fe1a5732ac06bbfcce88e1f1e7ed1cde6b4a6e2233a
Nov 03 12:22:39.458897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:39.458694    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn" event={"ID":"8c5fff96-cfca-4d24-8b0e-9b4091aa994f","Type":"ContainerStarted","Data":"3a29dff9ab02074ed49d3fe1a5732ac06bbfcce88e1f1e7ed1cde6b4a6e2233a"}
Nov 03 12:22:40.467176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:40.467146    4546 generic.go:356] "Generic (PLEG): container finished" podID="8c5fff96-cfca-4d24-8b0e-9b4091aa994f" containerID="a1042319bfe4f3b54461b8a007064532af90b8caca8d710a686bf4f72cfceff9" exitCode=0
Nov 03 12:22:40.467594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:40.467236    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn" event={"ID":"8c5fff96-cfca-4d24-8b0e-9b4091aa994f","Type":"ContainerDied","Data":"a1042319bfe4f3b54461b8a007064532af90b8caca8d710a686bf4f72cfceff9"}
Nov 03 12:22:41.626670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.626641    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:41.754504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.754436    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca\") pod \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") "
Nov 03 12:22:41.754642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.754621    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm\" (UniqueName: \"kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm\") pod \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") "
Nov 03 12:22:41.754683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.754660    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca\") pod \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\" (UID: \"8c5fff96-cfca-4d24-8b0e-9b4091aa994f\") "
Nov 03 12:22:41.755157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.755078    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca" pod "8c5fff96-cfca-4d24-8b0e-9b4091aa994f" (UID: "8c5fff96-cfca-4d24-8b0e-9b4091aa994f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:41.756656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.756626    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca" pod "8c5fff96-cfca-4d24-8b0e-9b4091aa994f" (UID: "8c5fff96-cfca-4d24-8b0e-9b4091aa994f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:22:41.756810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.756769    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm" pod "8c5fff96-cfca-4d24-8b0e-9b4091aa994f" (UID: "8c5fff96-cfca-4d24-8b0e-9b4091aa994f"). InnerVolumeSpecName "kube-api-access-jjksm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:41.856395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.856369    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjksm\" (UniqueName: \"kubernetes.io/projected/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-kube-api-access-jjksm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:41.856395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.856394    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:41.856553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:41.856404    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5fff96-cfca-4d24-8b0e-9b4091aa994f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:42.481151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:42.481124    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"
Nov 03 12:22:42.481301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:42.481159    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn" event={"ID":"8c5fff96-cfca-4d24-8b0e-9b4091aa994f","Type":"ContainerDied","Data":"3a29dff9ab02074ed49d3fe1a5732ac06bbfcce88e1f1e7ed1cde6b4a6e2233a"}
Nov 03 12:22:42.481301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:42.481207    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a29dff9ab02074ed49d3fe1a5732ac06bbfcce88e1f1e7ed1cde6b4a6e2233a"
Nov 03 12:22:49.472283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.472246    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"]
Nov 03 12:22:49.472689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.472471    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g" podUID="33b9d6fa-d74a-40d2-a7b0-853710870c0f" containerName="payload-processing" containerID="cri-o://16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4" gracePeriod=30
Nov 03 12:22:49.526068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.526037    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"]
Nov 03 12:22:49.526315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.526285    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4" podUID="b48fc8e4-7945-45eb-b810-281dfad016dd" containerName="payload-pre-processing" containerID="cri-o://b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a" gracePeriod=30
Nov 03 12:22:49.849478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.849446    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:22:49.867605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.867582    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:22:49.972832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.969647    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds\" (UniqueName: \"kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds\") pod \"b48fc8e4-7945-45eb-b810-281dfad016dd\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") "
Nov 03 12:22:49.972832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.969710    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv\" (UniqueName: \"kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv\") pod \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") "
Nov 03 12:22:49.972832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.969787    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume\") pod \"b48fc8e4-7945-45eb-b810-281dfad016dd\" (UID: \"b48fc8e4-7945-45eb-b810-281dfad016dd\") "
Nov 03 12:22:49.972832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.969855    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume\") pod \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\" (UID: \"33b9d6fa-d74a-40d2-a7b0-853710870c0f\") "
Nov 03 12:22:49.997821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.987997    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume" pod "b48fc8e4-7945-45eb-b810-281dfad016dd" (UID: "b48fc8e4-7945-45eb-b810-281dfad016dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:49.997821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:49.988854    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume" pod "33b9d6fa-d74a-40d2-a7b0-853710870c0f" (UID: "33b9d6fa-d74a-40d2-a7b0-853710870c0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:50.049331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.049292    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv" pod "33b9d6fa-d74a-40d2-a7b0-853710870c0f" (UID: "33b9d6fa-d74a-40d2-a7b0-853710870c0f"). InnerVolumeSpecName "kube-api-access-4jkpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:50.067948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.067910    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds" pod "b48fc8e4-7945-45eb-b810-281dfad016dd" (UID: "b48fc8e4-7945-45eb-b810-281dfad016dd"). InnerVolumeSpecName "kube-api-access-hr6ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:50.071367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.071338    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr6ds\" (UniqueName: \"kubernetes.io/projected/b48fc8e4-7945-45eb-b810-281dfad016dd-kube-api-access-hr6ds\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:50.071568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.071547    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jkpv\" (UniqueName: \"kubernetes.io/projected/33b9d6fa-d74a-40d2-a7b0-853710870c0f-kube-api-access-4jkpv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:50.071676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.071570    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48fc8e4-7945-45eb-b810-281dfad016dd-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:50.071676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.071584    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9d6fa-d74a-40d2-a7b0-853710870c0f-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:50.539942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.539909    4546 generic.go:356] "Generic (PLEG): container finished" podID="33b9d6fa-d74a-40d2-a7b0-853710870c0f" containerID="16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4" exitCode=0
Nov 03 12:22:50.540359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.539960    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"
Nov 03 12:22:50.540359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.539997    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g" event={"ID":"33b9d6fa-d74a-40d2-a7b0-853710870c0f","Type":"ContainerDied","Data":"16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4"}
Nov 03 12:22:50.540359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.540053    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g" event={"ID":"33b9d6fa-d74a-40d2-a7b0-853710870c0f","Type":"ContainerDied","Data":"3bd86e430756b6fc168653ed0b086fe18caac221b2da32f15b3a908d7caca282"}
Nov 03 12:22:50.540359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.540081    4546 scope.go:122] "RemoveContainer" containerID="16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4"
Nov 03 12:22:50.541733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.541713    4546 generic.go:356] "Generic (PLEG): container finished" podID="b48fc8e4-7945-45eb-b810-281dfad016dd" containerID="b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a" exitCode=0
Nov 03 12:22:50.541857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.541782    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"
Nov 03 12:22:50.541916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.541860    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4" event={"ID":"b48fc8e4-7945-45eb-b810-281dfad016dd","Type":"ContainerDied","Data":"b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a"}
Nov 03 12:22:50.541976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.541911    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4" event={"ID":"b48fc8e4-7945-45eb-b810-281dfad016dd","Type":"ContainerDied","Data":"f0305cdf39dcb578e66332091858ceb7b35b76424fb38bfe53b28878083d5575"}
Nov 03 12:22:50.552275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.552236    4546 scope.go:122] "RemoveContainer" containerID="16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4"
Nov 03 12:22:50.552559 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:50.552528    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4\": container with ID starting with 16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4 not found: ID does not exist" containerID="16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4"
Nov 03 12:22:50.552670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.552577    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4"} err="failed to get container status \"16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4\": rpc error: code = NotFound desc = could not find container \"16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4\": container with ID starting with 16a813addfaca195f7d1e84fd3cae641e3627924642c8cf52a341594c0b8d1e4 not found: ID does not exist"
Nov 03 12:22:50.552670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.552600    4546 scope.go:122] "RemoveContainer" containerID="b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a"
Nov 03 12:22:50.562813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.562625    4546 scope.go:122] "RemoveContainer" containerID="b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a"
Nov 03 12:22:50.563031 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:22:50.563012    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a\": container with ID starting with b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a not found: ID does not exist" containerID="b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a"
Nov 03 12:22:50.563087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.563036    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a"} err="failed to get container status \"b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a\": rpc error: code = NotFound desc = could not find container \"b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a\": container with ID starting with b454485a3454c36ee07149fe7fb772a9e8432760ff75642809659d6350f9052a not found: ID does not exist"
Nov 03 12:22:50.693451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.693423    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"]
Nov 03 12:22:50.699485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.699459    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e8d12-6dddc485f-d8mn4"]
Nov 03 12:22:50.800799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.800692    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"]
Nov 03 12:22:50.803711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:50.803688    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e8d12-79df7cbcd-6t79g"]
Nov 03 12:22:52.575683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:52.575651    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b9d6fa-d74a-40d2-a7b0-853710870c0f" path="/var/lib/kubelet/pods/33b9d6fa-d74a-40d2-a7b0-853710870c0f/volumes"
Nov 03 12:22:52.576237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:52.576220    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48fc8e4-7945-45eb-b810-281dfad016dd" path="/var/lib/kubelet/pods/b48fc8e4-7945-45eb-b810-281dfad016dd/volumes"
Nov 03 12:22:57.147275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:57.147231    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"]
Nov 03 12:22:57.147796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:57.147582    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy" containerID="cri-o://40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443" gracePeriod=30
Nov 03 12:22:59.983158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:59.983125    4546 scope.go:122] "RemoveContainer" containerID="123d713a70250fc1d99ebbcb71becbcfa92fd192147d28fc444a57bb12a7336d"
Nov 03 12:22:59.992700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:22:59.992675    4546 scope.go:122] "RemoveContainer" containerID="0d6927ffcdf4b79a3a55b243a4ce930f44bdace8020e12302abf677def4ffa94"
Nov 03 12:23:01.669284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:01.669249    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"]
Nov 03 12:23:01.767295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:01.767266    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"]
Nov 03 12:23:02.430469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.430446    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:23:02.546681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.546642    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.546896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.546790    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.546896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.546856    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.546914    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.546977    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547011    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547136    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547193    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547266    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data\") pod \"e70b937e-f686-4efd-9950-0aa472f0169b\" (UID: \"e70b937e-f686-4efd-9950-0aa472f0169b\") "
Nov 03 12:23:02.547518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547454    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:02.547709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547690    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:02.547812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547706    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:02.547812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547718    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:02.547962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.547941    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:02.548296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.548251    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.548296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.548273    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.548296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.548287    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70b937e-f686-4efd-9950-0aa472f0169b-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.548497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.548301    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.548497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.548313    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.549318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.549294    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:02.549619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.549599    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:02.549831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.549807    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:02.550715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.550690    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7" pod "e70b937e-f686-4efd-9950-0aa472f0169b" (UID: "e70b937e-f686-4efd-9950-0aa472f0169b"). InnerVolumeSpecName "kube-api-access-25gv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:02.633301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.633273    4546 generic.go:356] "Generic (PLEG): container finished" podID="e70b937e-f686-4efd-9950-0aa472f0169b" containerID="40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443" exitCode=0
Nov 03 12:23:02.633463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.633349    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"
Nov 03 12:23:02.633463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.633401    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" event={"ID":"e70b937e-f686-4efd-9950-0aa472f0169b","Type":"ContainerDied","Data":"40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443"}
Nov 03 12:23:02.633463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.633459    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2" event={"ID":"e70b937e-f686-4efd-9950-0aa472f0169b","Type":"ContainerDied","Data":"e37c7c32ed7959a875e762e2294ae6bf7dc10f896a91b550553f61acac416b4b"}
Nov 03 12:23:02.633621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.633490    4546 scope.go:122] "RemoveContainer" containerID="40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443"
Nov 03 12:23:02.644034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.644007    4546 scope.go:122] "RemoveContainer" containerID="4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"
Nov 03 12:23:02.650029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.650003    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25gv7\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-kube-api-access-25gv7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.650139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.650036    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70b937e-f686-4efd-9950-0aa472f0169b-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.650139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.650052    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70b937e-f686-4efd-9950-0aa472f0169b-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.650139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.650065    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70b937e-f686-4efd-9950-0aa472f0169b-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:02.653495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.653477    4546 scope.go:122] "RemoveContainer" containerID="40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443"
Nov 03 12:23:02.653744 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:02.653727    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443\": container with ID starting with 40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443 not found: ID does not exist" containerID="40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443"
Nov 03 12:23:02.653744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.653766    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443"} err="failed to get container status \"40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443\": rpc error: code = NotFound desc = could not find container \"40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443\": container with ID starting with 40f889a9e7be4a3e7ec6e7c0c178f73030dba0e247a86a53edc021013c690443 not found: ID does not exist"
Nov 03 12:23:02.653744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.653788    4546 scope.go:122] "RemoveContainer" containerID="4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"
Nov 03 12:23:02.654033 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:02.654016    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4\": container with ID starting with 4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4 not found: ID does not exist" containerID="4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"
Nov 03 12:23:02.654077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.654038    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4"} err="failed to get container status \"4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4\": rpc error: code = NotFound desc = could not find container \"4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4\": container with ID starting with 4589e86f8ab5db72f4d088c523d5e2861ffb00e1cd239c6c350243047ac533f4 not found: ID does not exist"
Nov 03 12:23:02.787944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.787910    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"]
Nov 03 12:23:02.791121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:02.791098    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e8d12-openshift-default-68b899ff7f-55hr2"]
Nov 03 12:23:04.580358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:04.580324    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" path="/var/lib/kubelet/pods/e70b937e-f686-4efd-9950-0aa472f0169b/volumes"
Nov 03 12:23:04.898577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:04.898498    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"]
Nov 03 12:23:04.898778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:04.898718    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5" podUID="ac5d16af-69e6-4486-a116-66891547be99" containerName="payload-processing" containerID="cri-o://cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693" gracePeriod=30
Nov 03 12:23:04.948570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:04.948545    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"]
Nov 03 12:23:04.948847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:04.948788    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr" podUID="e793873c-c9da-437b-bfa3-06a9be0ef4f8" containerName="payload-pre-processing" containerID="cri-o://85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd" gracePeriod=30
Nov 03 12:23:05.257872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.257851    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:23:05.260919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.260901    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:23:05.405106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.384360    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume\") pod \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") "
Nov 03 12:23:05.405106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.384467    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume\") pod \"ac5d16af-69e6-4486-a116-66891547be99\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") "
Nov 03 12:23:05.405106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.384672    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh\") pod \"ac5d16af-69e6-4486-a116-66891547be99\" (UID: \"ac5d16af-69e6-4486-a116-66891547be99\") "
Nov 03 12:23:05.405106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.384706    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d\" (UniqueName: \"kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d\") pod \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\" (UID: \"e793873c-c9da-437b-bfa3-06a9be0ef4f8\") "
Nov 03 12:23:05.405106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.392431    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume" pod "e793873c-c9da-437b-bfa3-06a9be0ef4f8" (UID: "e793873c-c9da-437b-bfa3-06a9be0ef4f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:05.454854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.450990    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d" pod "e793873c-c9da-437b-bfa3-06a9be0ef4f8" (UID: "e793873c-c9da-437b-bfa3-06a9be0ef4f8"). InnerVolumeSpecName "kube-api-access-c246d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:05.454854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.453978    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume" pod "ac5d16af-69e6-4486-a116-66891547be99" (UID: "ac5d16af-69e6-4486-a116-66891547be99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:05.462527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.462492    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh" pod "ac5d16af-69e6-4486-a116-66891547be99" (UID: "ac5d16af-69e6-4486-a116-66891547be99"). InnerVolumeSpecName "kube-api-access-rxxvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:05.486440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.486413    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c246d\" (UniqueName: \"kubernetes.io/projected/e793873c-c9da-437b-bfa3-06a9be0ef4f8-kube-api-access-c246d\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:05.486440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.486440    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e793873c-c9da-437b-bfa3-06a9be0ef4f8-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:05.486699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.486450    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d16af-69e6-4486-a116-66891547be99-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:05.486699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.486459    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/ac5d16af-69e6-4486-a116-66891547be99-kube-api-access-rxxvh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:05.656951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.656916    4546 generic.go:356] "Generic (PLEG): container finished" podID="e793873c-c9da-437b-bfa3-06a9be0ef4f8" containerID="85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd" exitCode=0
Nov 03 12:23:05.657394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.656975    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"
Nov 03 12:23:05.657394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.657017    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr" event={"ID":"e793873c-c9da-437b-bfa3-06a9be0ef4f8","Type":"ContainerDied","Data":"85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd"}
Nov 03 12:23:05.657394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.657066    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr" event={"ID":"e793873c-c9da-437b-bfa3-06a9be0ef4f8","Type":"ContainerDied","Data":"710638a0f5b138e003e4e7cc5fc5bf2e0f0ef5fe29abe92d109aac0b96c1ebb5"}
Nov 03 12:23:05.657394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.657088    4546 scope.go:122] "RemoveContainer" containerID="85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd"
Nov 03 12:23:05.658657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.658633    4546 generic.go:356] "Generic (PLEG): container finished" podID="ac5d16af-69e6-4486-a116-66891547be99" containerID="cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693" exitCode=0
Nov 03 12:23:05.658811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.658700    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"
Nov 03 12:23:05.658811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.658723    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5" event={"ID":"ac5d16af-69e6-4486-a116-66891547be99","Type":"ContainerDied","Data":"cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693"}
Nov 03 12:23:05.658811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.658799    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5" event={"ID":"ac5d16af-69e6-4486-a116-66891547be99","Type":"ContainerDied","Data":"400ab58aa31ea1876d19fe537342d670d7ac814c7023e9596741e1e2a3db263d"}
Nov 03 12:23:05.667118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.667102    4546 scope.go:122] "RemoveContainer" containerID="85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd"
Nov 03 12:23:05.667391 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:05.667369    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd\": container with ID starting with 85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd not found: ID does not exist" containerID="85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd"
Nov 03 12:23:05.667499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.667397    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd"} err="failed to get container status \"85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd\": rpc error: code = NotFound desc = could not find container \"85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd\": container with ID starting with 85229d3d2d2463bc7ca9db7b138a513e5cdb8727b01c6e357d4ab6373292bbbd not found: ID does not exist"
Nov 03 12:23:05.667499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.667414    4546 scope.go:122] "RemoveContainer" containerID="cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693"
Nov 03 12:23:05.676611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.676596    4546 scope.go:122] "RemoveContainer" containerID="cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693"
Nov 03 12:23:05.676903 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:05.676879    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693\": container with ID starting with cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693 not found: ID does not exist" containerID="cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693"
Nov 03 12:23:05.676963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.676910    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693"} err="failed to get container status \"cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693\": rpc error: code = NotFound desc = could not find container \"cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693\": container with ID starting with cee301ac3bc7f804ad534c60b590325e8e9daa8fd42f3138b8f17f1c227b8693 not found: ID does not exist"
Nov 03 12:23:05.809716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.809693    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"]
Nov 03 12:23:05.812719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.812698    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45b1b7-556bf6499f-wz2cr"]
Nov 03 12:23:05.911402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.911374    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"]
Nov 03 12:23:05.914340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:05.914315    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45b1b7-84896f44bb-b24n5"]
Nov 03 12:23:06.574650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:06.574611    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5d16af-69e6-4486-a116-66891547be99" path="/var/lib/kubelet/pods/ac5d16af-69e6-4486-a116-66891547be99/volumes"
Nov 03 12:23:06.575209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:06.575185    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e793873c-c9da-437b-bfa3-06a9be0ef4f8" path="/var/lib/kubelet/pods/e793873c-c9da-437b-bfa3-06a9be0ef4f8/volumes"
Nov 03 12:23:06.669860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:06.669825    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb" podUID="4402fad5-d02e-4966-82fb-af42f96142b0" containerName="payload-processing" containerID="cri-o://ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d" gracePeriod=25
Nov 03 12:23:06.768059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:06.768015    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt" podUID="d282d4ac-1ef8-412b-9d54-e91ec08a25a1" containerName="payload-pre-processing" containerID="cri-o://ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4" gracePeriod=25
Nov 03 12:23:06.953626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:06.953586    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:23:07.004784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.004736    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n\" (UniqueName: \"kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n\") pod \"4402fad5-d02e-4966-82fb-af42f96142b0\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") "
Nov 03 12:23:07.004975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.004872    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume\") pod \"4402fad5-d02e-4966-82fb-af42f96142b0\" (UID: \"4402fad5-d02e-4966-82fb-af42f96142b0\") "
Nov 03 12:23:07.005470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.005439    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume" pod "4402fad5-d02e-4966-82fb-af42f96142b0" (UID: "4402fad5-d02e-4966-82fb-af42f96142b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:07.007967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.007940    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n" pod "4402fad5-d02e-4966-82fb-af42f96142b0" (UID: "4402fad5-d02e-4966-82fb-af42f96142b0"). InnerVolumeSpecName "kube-api-access-z2m2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:07.056715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.056692    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:23:07.106491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.106411    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume\") pod \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") "
Nov 03 12:23:07.106491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.106476    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8\" (UniqueName: \"kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8\") pod \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\" (UID: \"d282d4ac-1ef8-412b-9d54-e91ec08a25a1\") "
Nov 03 12:23:07.106842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.106819    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume" pod "d282d4ac-1ef8-412b-9d54-e91ec08a25a1" (UID: "d282d4ac-1ef8-412b-9d54-e91ec08a25a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:07.106928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.106880    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4402fad5-d02e-4966-82fb-af42f96142b0-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:07.106928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.106894    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2m2n\" (UniqueName: \"kubernetes.io/projected/4402fad5-d02e-4966-82fb-af42f96142b0-kube-api-access-z2m2n\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:07.108728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.108701    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8" pod "d282d4ac-1ef8-412b-9d54-e91ec08a25a1" (UID: "d282d4ac-1ef8-412b-9d54-e91ec08a25a1"). InnerVolumeSpecName "kube-api-access-kl8z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:07.208939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.208906    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:07.208939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.208940    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl8z8\" (UniqueName: \"kubernetes.io/projected/d282d4ac-1ef8-412b-9d54-e91ec08a25a1-kube-api-access-kl8z8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:07.652690 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:07.652619    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:23:07.674977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.674952    4546 generic.go:356] "Generic (PLEG): container finished" podID="d282d4ac-1ef8-412b-9d54-e91ec08a25a1" containerID="ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4" exitCode=0
Nov 03 12:23:07.675405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.675003    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"
Nov 03 12:23:07.675405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.675022    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt" event={"ID":"d282d4ac-1ef8-412b-9d54-e91ec08a25a1","Type":"ContainerDied","Data":"ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4"}
Nov 03 12:23:07.675405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.675076    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt" event={"ID":"d282d4ac-1ef8-412b-9d54-e91ec08a25a1","Type":"ContainerDied","Data":"24a934b78c20fc3958ff06dbb39c32ffb618ce06603dc71c0ac03b164a454841"}
Nov 03 12:23:07.675405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.675102    4546 scope.go:122] "RemoveContainer" containerID="ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4"
Nov 03 12:23:07.680671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.680551    4546 generic.go:356] "Generic (PLEG): container finished" podID="4402fad5-d02e-4966-82fb-af42f96142b0" containerID="ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d" exitCode=0
Nov 03 12:23:07.680671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.680613    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb" event={"ID":"4402fad5-d02e-4966-82fb-af42f96142b0","Type":"ContainerDied","Data":"ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d"}
Nov 03 12:23:07.680671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.680646    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb" event={"ID":"4402fad5-d02e-4966-82fb-af42f96142b0","Type":"ContainerDied","Data":"40d602bd9e500776cf9f680bf1b7f7e870730e56aff1d316a79121ea3bd48fb0"}
Nov 03 12:23:07.681043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.680701    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"
Nov 03 12:23:07.689622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.689604    4546 scope.go:122] "RemoveContainer" containerID="ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4"
Nov 03 12:23:07.689927 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:07.689907    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4\": container with ID starting with ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4 not found: ID does not exist" containerID="ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4"
Nov 03 12:23:07.690011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.689938    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4"} err="failed to get container status \"ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4\": rpc error: code = NotFound desc = could not find container \"ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4\": container with ID starting with ffdddb19093acff9f825ce666ef92c612bdf84d478802e5da9e75fba1867dbb4 not found: ID does not exist"
Nov 03 12:23:07.690011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.689961    4546 scope.go:122] "RemoveContainer" containerID="ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d"
Nov 03 12:23:07.699988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.699966    4546 scope.go:122] "RemoveContainer" containerID="ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d"
Nov 03 12:23:07.700254 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:07.700232    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d\": container with ID starting with ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d not found: ID does not exist" containerID="ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d"
Nov 03 12:23:07.700318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.700261    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d"} err="failed to get container status \"ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d\": rpc error: code = NotFound desc = could not find container \"ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d\": container with ID starting with ad2bbc57b93a4e3610d4a37796f7a2f8806f396d74bd49a698c72688d457a59d not found: ID does not exist"
Nov 03 12:23:07.829712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.829682    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"]
Nov 03 12:23:07.875738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.875707    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-784cd744d5-kd9pb"]
Nov 03 12:23:07.933673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.933587    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"]
Nov 03 12:23:07.936730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:07.936710    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-69946cf795-p26mt"]
Nov 03 12:23:08.166850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:08.166813    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"]
Nov 03 12:23:08.167077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:08.167023    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" containerID="cri-o://3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f" gracePeriod=30
Nov 03 12:23:08.574586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:08.574557    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4402fad5-d02e-4966-82fb-af42f96142b0" path="/var/lib/kubelet/pods/4402fad5-d02e-4966-82fb-af42f96142b0/volumes"
Nov 03 12:23:08.574971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:08.574955    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d282d4ac-1ef8-412b-9d54-e91ec08a25a1" path="/var/lib/kubelet/pods/d282d4ac-1ef8-412b-9d54-e91ec08a25a1/volumes"
Nov 03 12:23:11.039984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:11.039938    4546 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:23:11.040374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:11.039990    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:23:13.427297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.427259    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"]
Nov 03 12:23:13.428201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428175    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ac5d16af-69e6-4486-a116-66891547be99" containerName="payload-processing"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428203    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5d16af-69e6-4486-a116-66891547be99" containerName="payload-processing"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428232    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4402fad5-d02e-4966-82fb-af42f96142b0" containerName="payload-processing"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428241    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4402fad5-d02e-4966-82fb-af42f96142b0" containerName="payload-processing"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428257    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428265    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428308    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8c5fff96-cfca-4d24-8b0e-9b4091aa994f" containerName="revoke-keys"
Nov 03 12:23:13.428316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428317    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5fff96-cfca-4d24-8b0e-9b4091aa994f" containerName="revoke-keys"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428330    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e793873c-c9da-437b-bfa3-06a9be0ef4f8" containerName="payload-pre-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428338    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e793873c-c9da-437b-bfa3-06a9be0ef4f8" containerName="payload-pre-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428358    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33b9d6fa-d74a-40d2-a7b0-853710870c0f" containerName="payload-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428366    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b9d6fa-d74a-40d2-a7b0-853710870c0f" containerName="payload-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428382    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d282d4ac-1ef8-412b-9d54-e91ec08a25a1" containerName="payload-pre-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428390    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d282d4ac-1ef8-412b-9d54-e91ec08a25a1" containerName="payload-pre-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428400    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b48fc8e4-7945-45eb-b810-281dfad016dd" containerName="payload-pre-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428407    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48fc8e4-7945-45eb-b810-281dfad016dd" containerName="payload-pre-processing"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428420    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428427    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy"
Nov 03 12:23:13.428570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428572    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b9d6fa-d74a-40d2-a7b0-853710870c0f" containerName="payload-processing"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428590    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d282d4ac-1ef8-412b-9d54-e91ec08a25a1" containerName="payload-pre-processing"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428599    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48fc8e4-7945-45eb-b810-281dfad016dd" containerName="payload-pre-processing"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428609    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428620    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5d16af-69e6-4486-a116-66891547be99" containerName="payload-processing"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428631    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4402fad5-d02e-4966-82fb-af42f96142b0" containerName="payload-processing"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428641    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70b937e-f686-4efd-9950-0aa472f0169b" containerName="istio-proxy"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428656    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5fff96-cfca-4d24-8b0e-9b4091aa994f" containerName="revoke-keys"
Nov 03 12:23:13.428968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.428666    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e793873c-c9da-437b-bfa3-06a9be0ef4f8" containerName="payload-pre-processing"
Nov 03 12:23:13.451977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.435234    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.485295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.485262    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c8ee66-openshift-default-dockercfg-4l8r5\""
Nov 03 12:23:13.536116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.536012    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"]
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619201    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnxc\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619262    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619297    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619349    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619391    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619442    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619496    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619562    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.647774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.619623    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.661207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.655290    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.721480    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.721589    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.721669    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.721734    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.721951    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnxc\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.721995    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.722028    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.722099    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.722142    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.722574    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.723037    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.724053    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.724288    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.724511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.724421    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.728414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.728385    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.730638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.730510    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.759818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.759787    4546 generic.go:356] "Generic (PLEG): container finished" podID="0a095a5f-8680-4b3c-a196-5db07230879f" containerID="3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f" exitCode=0
Nov 03 12:23:13.760000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.759854    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"
Nov 03 12:23:13.760000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.759983    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" event={"ID":"0a095a5f-8680-4b3c-a196-5db07230879f","Type":"ContainerDied","Data":"3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f"}
Nov 03 12:23:13.760121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.760033    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68" event={"ID":"0a095a5f-8680-4b3c-a196-5db07230879f","Type":"ContainerDied","Data":"3a65ab7b27c1f5da9c54734974240d2dc2f9a520d04d7712777c6cb3fb5157de"}
Nov 03 12:23:13.760121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.760065    4546 scope.go:122] "RemoveContainer" containerID="3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f"
Nov 03 12:23:13.777592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.777559    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.778388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.778361    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnxc\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc\") pod \"e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") " pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:13.780817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.780795    4546 scope.go:122] "RemoveContainer" containerID="3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f"
Nov 03 12:23:13.781192 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:13.781165    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f\": container with ID starting with 3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f not found: ID does not exist" containerID="3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f"
Nov 03 12:23:13.781257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.781208    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f"} err="failed to get container status \"3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f\": rpc error: code = NotFound desc = could not find container \"3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f\": container with ID starting with 3b5786b1bf55f5ad1f00a4b7d2cbe3f3958077cc04617c7bcae31fc5a09f822f not found: ID does not exist"
Nov 03 12:23:13.823625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.823586    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.823829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.823676    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.823829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.823712    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.823829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.823774    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.824037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.823832    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.824037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.823981    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.824153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.824067    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.824153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.824133    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.824264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.824176    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket\") pod \"0a095a5f-8680-4b3c-a196-5db07230879f\" (UID: \"0a095a5f-8680-4b3c-a196-5db07230879f\") "
Nov 03 12:23:13.825058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.825029    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:13.825568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.825545    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:13.826028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.826007    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:13.826671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.826643    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:13.826882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.826861    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:13.830169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.830135    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:13.830695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.830668    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:13.832265 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.832237    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:13.836973 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.836942    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l" pod "0a095a5f-8680-4b3c-a196-5db07230879f" (UID: "0a095a5f-8680-4b3c-a196-5db07230879f"). InnerVolumeSpecName "kube-api-access-vp99l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:13.926268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926219    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp99l\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-kube-api-access-vp99l\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926265    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926282    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926299    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a095a5f-8680-4b3c-a196-5db07230879f-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926313    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926326    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a095a5f-8680-4b3c-a196-5db07230879f-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926339    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926352    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a095a5f-8680-4b3c-a196-5db07230879f-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:13.926557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:13.926367    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a095a5f-8680-4b3c-a196-5db07230879f-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.057832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:14.057644    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:14.194425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:14.194213    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"]
Nov 03 12:23:14.198382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:14.198347    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-94q68"]
Nov 03 12:23:14.737972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:14.721240    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" path="/var/lib/kubelet/pods/0a095a5f-8680-4b3c-a196-5db07230879f/volumes"
Nov 03 12:23:15.018070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.016282    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"]
Nov 03 12:23:15.018070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.017431    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:23:15.018070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.017504    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:23:15.018070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.017545    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:23:15.591783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.576531    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:15.641331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.631029    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:15.671222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.631068    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Nov 03 12:23:15.695229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:15.657847    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.145789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.145709    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 12:23:16.244775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.241439    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" event={"ID":"b90b2734-457b-454d-b91d-d10560505147","Type":"ContainerStarted","Data":"16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"}
Nov 03 12:23:16.244775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.241510    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" event={"ID":"b90b2734-457b-454d-b91d-d10560505147","Type":"ContainerStarted","Data":"a1c0134100160c8362faf8e0d47781d044b75b7bdb556dd5bd11aa17059fde1a"}
Nov 03 12:23:16.303778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.287195    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.362794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.361922    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ltzvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:23:16.362794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.362000    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.650783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.649166    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.707873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.707828    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.708386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.708334    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:23:16.708927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.708881    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.709202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.709185    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:23:16.743617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.743587    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 12:23:16.801712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.801653    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:23:16.832775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.828092    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podStartSLOduration=3.8280707290000002 podStartE2EDuration="3.828070729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:16.459080942 +0000 UTC m=+1938.030664856" watchObservedRunningTime="2026-11-03 12:23:16.828070729 +0000 UTC m=+1938.399654642"
Nov 03 12:23:16.996876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.996846    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"]
Nov 03 12:23:16.997464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.997448    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy"
Nov 03 12:23:16.997539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.997466    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy"
Nov 03 12:23:16.997580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:16.997562    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a095a5f-8680-4b3c-a196-5db07230879f" containerName="istio-proxy"
Nov 03 12:23:17.001368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.001346    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.054059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.054012    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2f3eb2\""
Nov 03 12:23:17.054372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.054344    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.054663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.054641    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mjw\" (UniqueName: \"kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.055170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.055147    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"]
Nov 03 12:23:17.058185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.058164    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:17.071788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.067040    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:17.135283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.135218    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2f3eb2-dockercfg-vptfr\""
Nov 03 12:23:17.156078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.156045    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mjw\" (UniqueName: \"kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.156527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.156144    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.156888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.156864    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.234772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.233095    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mjw\" (UniqueName: \"kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw\") pod \"payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.248866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.248789    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:17.249856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.249818    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:23:17.344552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.344526    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:17.741615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.741583    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"]
Nov 03 12:23:17.742337 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:17.742303    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6bc69a_bfa7_4158_8318_d7573a40b1b0.slice/crio-f32742b14063e6ec495abc2453eb60b73bb2a0e81afdff5e0ebf20ce3af8a99c WatchSource:0}: Error finding container f32742b14063e6ec495abc2453eb60b73bb2a0e81afdff5e0ebf20ce3af8a99c: Status 404 returned error can't find the container with id f32742b14063e6ec495abc2453eb60b73bb2a0e81afdff5e0ebf20ce3af8a99c
Nov 03 12:23:17.843288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.843242    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"]
Nov 03 12:23:17.849405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.849386    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:17.865773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.865727    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:17.866003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.865981    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75fz\" (UniqueName: \"kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz\") pod \"payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:17.938883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.938842    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"]
Nov 03 12:23:17.967153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.967110    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75fz\" (UniqueName: \"kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz\") pod \"payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:17.967349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.967214    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:17.967907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:17.967885    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:18.034636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.034549    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75fz\" (UniqueName: \"kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz\") pod \"payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:18.165223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.165190    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:18.278825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.278790    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz" event={"ID":"6d6bc69a-bfa7-4158-8318-d7573a40b1b0","Type":"ContainerStarted","Data":"017347fd6ff7f4aeef4d374c10fec573ecfad99d6863b571bc6f3c89241dfd47"}
Nov 03 12:23:18.278978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.278856    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz" event={"ID":"6d6bc69a-bfa7-4158-8318-d7573a40b1b0","Type":"ContainerStarted","Data":"f32742b14063e6ec495abc2453eb60b73bb2a0e81afdff5e0ebf20ce3af8a99c"}
Nov 03 12:23:18.278978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.278890    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:18.386856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.386728    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz" podStartSLOduration=2.386715104 podStartE2EDuration="2.386715104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:18.384546874 +0000 UTC m=+1939.956130786" watchObservedRunningTime="2026-11-03 12:23:18.386715104 +0000 UTC m=+1939.958299042"
Nov 03 12:23:18.568790 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:18.568706    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0592ed8_db55_4ca6_b7fa_7fd0608d59ef.slice/crio-b47ceca6d8a8836889d0114f709e0306efc186a49e80a4001d2613f8767d05a2 WatchSource:0}: Error finding container b47ceca6d8a8836889d0114f709e0306efc186a49e80a4001d2613f8767d05a2: Status 404 returned error can't find the container with id b47ceca6d8a8836889d0114f709e0306efc186a49e80a4001d2613f8767d05a2
Nov 03 12:23:18.577454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:18.577427    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"]
Nov 03 12:23:19.290408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:19.290372    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" event={"ID":"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef","Type":"ContainerStarted","Data":"5973ac90ced3bfb43ac07a84b6924c9676e3fbb10b78a9ea472e7e6bd289ccbd"}
Nov 03 12:23:19.290943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:19.290493    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" event={"ID":"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef","Type":"ContainerStarted","Data":"b47ceca6d8a8836889d0114f709e0306efc186a49e80a4001d2613f8767d05a2"}
Nov 03 12:23:19.290943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:19.290882    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:19.398634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:19.398578    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" podStartSLOduration=2.398561747 podStartE2EDuration="2.398561747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:19.397046404 +0000 UTC m=+1940.968630341" watchObservedRunningTime="2026-11-03 12:23:19.398561747 +0000 UTC m=+1940.970145660"
Nov 03 12:23:22.575027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.575000    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"]
Nov 03 12:23:22.579179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.579161    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.632818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.632760    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"]
Nov 03 12:23:22.657605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.651688    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7767n\" (UniqueName: \"kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n\") pod \"payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.657605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.651813    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.694278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.687011    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c8ee66-dockercfg-272br\""
Nov 03 12:23:22.694278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.687150    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c8ee66\""
Nov 03 12:23:22.757199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.756128    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7767n\" (UniqueName: \"kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n\") pod \"payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.757199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.756201    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.757199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.756863    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.809900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.809869    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7767n\" (UniqueName: \"kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n\") pod \"payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:22.894037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:22.893957    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:23.050892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:23.050853    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:23:23.051055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:23.050901    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:23.262344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:23.262315    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"]
Nov 03 12:23:23.264082 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:23.264053    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d8c249_c37f_42c4_a0f9_2e613b05e8e7.slice/crio-fdbdff9332193fd01addb0206d263f4682d47afe0391dedc9c3f4edfd8af9325 WatchSource:0}: Error finding container fdbdff9332193fd01addb0206d263f4682d47afe0391dedc9c3f4edfd8af9325: Status 404 returned error can't find the container with id fdbdff9332193fd01addb0206d263f4682d47afe0391dedc9c3f4edfd8af9325
Nov 03 12:23:23.338481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:23.338448    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f" event={"ID":"86d8c249-c37f-42c4-a0f9-2e613b05e8e7","Type":"ContainerStarted","Data":"fdbdff9332193fd01addb0206d263f4682d47afe0391dedc9c3f4edfd8af9325"}
Nov 03 12:23:24.346655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.346616    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f" event={"ID":"86d8c249-c37f-42c4-a0f9-2e613b05e8e7","Type":"ContainerStarted","Data":"31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92"}
Nov 03 12:23:24.347173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.346692    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:24.348341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.348320    4546 generic.go:356] "Generic (PLEG): container finished" podID="867be5c5-4007-4c7b-b4f2-b89440379906" containerID="aa284368b47faaeced64cf6cc6d1b1346da47749e86d88fb80be801d3bd65f01" exitCode=0
Nov 03 12:23:24.348441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.348393    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" event={"ID":"867be5c5-4007-4c7b-b4f2-b89440379906","Type":"ContainerDied","Data":"aa284368b47faaeced64cf6cc6d1b1346da47749e86d88fb80be801d3bd65f01"}
Nov 03 12:23:24.348772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.348738    4546 scope.go:122] "RemoveContainer" containerID="aa284368b47faaeced64cf6cc6d1b1346da47749e86d88fb80be801d3bd65f01"
Nov 03 12:23:24.350278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.350254    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:23:24.350348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.350317    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:23:24.350385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.350347    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:23:24.564985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:24.564929    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f" podStartSLOduration=2.564906558 podStartE2EDuration="2.564906558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:24.454993818 +0000 UTC m=+1946.026577729" watchObservedRunningTime="2026-11-03 12:23:24.564906558 +0000 UTC m=+1946.136490467"
Nov 03 12:23:25.357065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:25.357033    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" event={"ID":"867be5c5-4007-4c7b-b4f2-b89440379906","Type":"ContainerStarted","Data":"d30d4dd0a9ded501937b541b9d522827ca789e8014ef8d4e1c77df86e8db513a"}
Nov 03 12:23:25.386860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:25.386838    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:23:25.387145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:25.387123    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 12:23:25.387196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:25.387175    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 12:23:26.386719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:26.386686    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 12:23:26.387238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:26.386733    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 12:23:27.387170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:27.387137    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 12:23:27.387559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:27.387186    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 12:23:28.387226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:28.387186    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 12:23:28.387603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:28.387250    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 12:23:29.292695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:29.292664    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:23:29.387515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:29.387486    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 12:23:29.387975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:29.387541    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 12:23:30.386954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:30.386917    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 12:23:30.387135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:30.386987    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 12:23:31.310536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:31.310511    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:23:31.391294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:31.391263    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:23:31.391566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:31.391550    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:23:31.392281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:31.392262    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:23:35.361037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:35.361007    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:23:37.657918 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:37.657882    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:23:43.037212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.037170    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"]
Nov 03 12:23:43.037796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.037370    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5" podUID="e23e16d8-2fa1-45bf-b110-02a0930ffd0f" containerName="payload-processing" containerID="cri-o://92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58" gracePeriod=30
Nov 03 12:23:43.137635 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.137598    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"]
Nov 03 12:23:43.138259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.138222    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr" podUID="3c3846e6-c03e-40e1-8551-34edbdd379b6" containerName="payload-pre-processing" containerID="cri-o://f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21" gracePeriod=30
Nov 03 12:23:43.337583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.337553    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:23:43.407100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.407078    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:23:43.437114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.437089    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume\") pod \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") "
Nov 03 12:23:43.437236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.437225    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg\" (UniqueName: \"kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg\") pod \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\" (UID: \"e23e16d8-2fa1-45bf-b110-02a0930ffd0f\") "
Nov 03 12:23:43.437430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.437409    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume" pod "e23e16d8-2fa1-45bf-b110-02a0930ffd0f" (UID: "e23e16d8-2fa1-45bf-b110-02a0930ffd0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:43.437689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.437674    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.439368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.439347    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg" pod "e23e16d8-2fa1-45bf-b110-02a0930ffd0f" (UID: "e23e16d8-2fa1-45bf-b110-02a0930ffd0f"). InnerVolumeSpecName "kube-api-access-r7slg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:43.474523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.474486    4546 generic.go:356] "Generic (PLEG): container finished" podID="3c3846e6-c03e-40e1-8551-34edbdd379b6" containerID="f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21" exitCode=0
Nov 03 12:23:43.474730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.474532    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"
Nov 03 12:23:43.474730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.474580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr" event={"ID":"3c3846e6-c03e-40e1-8551-34edbdd379b6","Type":"ContainerDied","Data":"f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21"}
Nov 03 12:23:43.474730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.474651    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr" event={"ID":"3c3846e6-c03e-40e1-8551-34edbdd379b6","Type":"ContainerDied","Data":"9e3c1fd1ed2b937d8c6654f5a22ad4ec41990ffd6b3dc6591e5326b3a31d2447"}
Nov 03 12:23:43.474730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.474685    4546 scope.go:122] "RemoveContainer" containerID="f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21"
Nov 03 12:23:43.476273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.476250    4546 generic.go:356] "Generic (PLEG): container finished" podID="e23e16d8-2fa1-45bf-b110-02a0930ffd0f" containerID="92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58" exitCode=0
Nov 03 12:23:43.476385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.476304    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"
Nov 03 12:23:43.476385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.476336    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5" event={"ID":"e23e16d8-2fa1-45bf-b110-02a0930ffd0f","Type":"ContainerDied","Data":"92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58"}
Nov 03 12:23:43.476521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.476399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5" event={"ID":"e23e16d8-2fa1-45bf-b110-02a0930ffd0f","Type":"ContainerDied","Data":"9b95db4eaf8a4882ee8058242ec5b8a4c4804e0941458f328e926caf1d982e03"}
Nov 03 12:23:43.490795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.488359    4546 scope.go:122] "RemoveContainer" containerID="f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21"
Nov 03 12:23:43.490795 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:43.488869    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21\": container with ID starting with f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21 not found: ID does not exist" containerID="f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21"
Nov 03 12:23:43.490795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.488938    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21"} err="failed to get container status \"f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21\": rpc error: code = NotFound desc = could not find container \"f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21\": container with ID starting with f92a8168968d43a37810d2525e72cc9459badc73c7c06a83489cbaf3c75f3a21 not found: ID does not exist"
Nov 03 12:23:43.490795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.488965    4546 scope.go:122] "RemoveContainer" containerID="92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58"
Nov 03 12:23:43.501397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.501379    4546 scope.go:122] "RemoveContainer" containerID="92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58"
Nov 03 12:23:43.501673 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:43.501651    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58\": container with ID starting with 92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58 not found: ID does not exist" containerID="92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58"
Nov 03 12:23:43.501803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.501679    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58"} err="failed to get container status \"92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58\": rpc error: code = NotFound desc = could not find container \"92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58\": container with ID starting with 92717f80bb8d9276a3dcf03de4de63067ca1e9a53dcdb905a584be27631e1a58 not found: ID does not exist"
Nov 03 12:23:43.545782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.542246    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume\") pod \"3c3846e6-c03e-40e1-8551-34edbdd379b6\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") "
Nov 03 12:23:43.545782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.542549    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t\" (UniqueName: \"kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t\") pod \"3c3846e6-c03e-40e1-8551-34edbdd379b6\" (UID: \"3c3846e6-c03e-40e1-8551-34edbdd379b6\") "
Nov 03 12:23:43.545782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.543024    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7slg\" (UniqueName: \"kubernetes.io/projected/e23e16d8-2fa1-45bf-b110-02a0930ffd0f-kube-api-access-r7slg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.570831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.554986    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume" pod "3c3846e6-c03e-40e1-8551-34edbdd379b6" (UID: "3c3846e6-c03e-40e1-8551-34edbdd379b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:43.611773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.611185    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t" pod "3c3846e6-c03e-40e1-8551-34edbdd379b6" (UID: "3c3846e6-c03e-40e1-8551-34edbdd379b6"). InnerVolumeSpecName "kube-api-access-2gm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:43.644658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.644630    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gm9t\" (UniqueName: \"kubernetes.io/projected/3c3846e6-c03e-40e1-8551-34edbdd379b6-kube-api-access-2gm9t\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.644658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.644655    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3846e6-c03e-40e1-8551-34edbdd379b6-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:43.703789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.703732    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"]
Nov 03 12:23:43.707495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.707466    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e151c-76755b6458-778z5"]
Nov 03 12:23:43.889625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.889534    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"]
Nov 03 12:23:43.892614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:43.892590    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e151c-5c498655dc-mkntr"]
Nov 03 12:23:44.573977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:44.573942    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3846e6-c03e-40e1-8551-34edbdd379b6" path="/var/lib/kubelet/pods/3c3846e6-c03e-40e1-8551-34edbdd379b6/volumes"
Nov 03 12:23:44.574368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:44.574337    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23e16d8-2fa1-45bf-b110-02a0930ffd0f" path="/var/lib/kubelet/pods/e23e16d8-2fa1-45bf-b110-02a0930ffd0f/volumes"
Nov 03 12:23:45.677572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.677542    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"]
Nov 03 12:23:45.678178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.678162    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e23e16d8-2fa1-45bf-b110-02a0930ffd0f" containerName="payload-processing"
Nov 03 12:23:45.678227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.678179    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23e16d8-2fa1-45bf-b110-02a0930ffd0f" containerName="payload-processing"
Nov 03 12:23:45.678227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.678189    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3c3846e6-c03e-40e1-8551-34edbdd379b6" containerName="payload-pre-processing"
Nov 03 12:23:45.678227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.678194    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3846e6-c03e-40e1-8551-34edbdd379b6" containerName="payload-pre-processing"
Nov 03 12:23:45.678354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.678310    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23e16d8-2fa1-45bf-b110-02a0930ffd0f" containerName="payload-processing"
Nov 03 12:23:45.678354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.678326    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3846e6-c03e-40e1-8551-34edbdd379b6" containerName="payload-pre-processing"
Nov 03 12:23:45.683269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.683250    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.742776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.742200    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"]
Nov 03 12:23:45.776566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.776518    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxph\" (UniqueName: \"kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph\") pod \"payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.776839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.776580    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.798907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.798063    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d25504-dockercfg-mlpx2\""
Nov 03 12:23:45.798907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.798182    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d25504\""
Nov 03 12:23:45.842388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.842351    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"]
Nov 03 12:23:45.846562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.846541    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:45.877827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.877794    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxph\" (UniqueName: \"kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph\") pod \"payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.877994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.877838    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.878546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.878527    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.899682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.899648    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"]
Nov 03 12:23:45.931045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.930968    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxph\" (UniqueName: \"kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph\") pod \"payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:45.979201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.979167    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:45.979384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.979234    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhvw\" (UniqueName: \"kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw\") pod \"payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:45.997001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:45.996968    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:46.081279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.080383    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:46.081279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.080446    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhvw\" (UniqueName: \"kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw\") pod \"payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:46.081518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.081456    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:46.134025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.133998    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhvw\" (UniqueName: \"kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw\") pod \"payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:46.162054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.162028    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:46.366517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.366487    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"]
Nov 03 12:23:46.367488 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:46.367451    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb1fdc7_db8f_4c09_8c29_f75e7cf301bb.slice/crio-0a2ae498a71982e128418aada21566bfe3e3809caec3d3622d0bd204e5661e4b WatchSource:0}: Error finding container 0a2ae498a71982e128418aada21566bfe3e3809caec3d3622d0bd204e5661e4b: Status 404 returned error can't find the container with id 0a2ae498a71982e128418aada21566bfe3e3809caec3d3622d0bd204e5661e4b
Nov 03 12:23:46.501092 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.500996    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft" event={"ID":"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb","Type":"ContainerStarted","Data":"a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16"}
Nov 03 12:23:46.501092 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.501048    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft" event={"ID":"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb","Type":"ContainerStarted","Data":"0a2ae498a71982e128418aada21566bfe3e3809caec3d3622d0bd204e5661e4b"}
Nov 03 12:23:46.501329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.501131    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:46.529278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.529247    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"]
Nov 03 12:23:46.529867 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:46.529841    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee614dc0_0626_4c29_9952_c0fc0c7ead47.slice/crio-77df98855940a98a24078fc182556c2875bbc4770bdcac8637d01eb654f3aea4 WatchSource:0}: Error finding container 77df98855940a98a24078fc182556c2875bbc4770bdcac8637d01eb654f3aea4: Status 404 returned error can't find the container with id 77df98855940a98a24078fc182556c2875bbc4770bdcac8637d01eb654f3aea4
Nov 03 12:23:46.608428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:46.608371    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft" podStartSLOduration=1.6083578699999999 podStartE2EDuration="1.60835787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:46.606986995 +0000 UTC m=+1968.178570928" watchObservedRunningTime="2026-11-03 12:23:46.60835787 +0000 UTC m=+1968.179941779"
Nov 03 12:23:47.508824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:47.508783    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9" event={"ID":"ee614dc0-0626-4c29-9952-c0fc0c7ead47","Type":"ContainerStarted","Data":"dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf"}
Nov 03 12:23:47.509263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:47.508847    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9" event={"ID":"ee614dc0-0626-4c29-9952-c0fc0c7ead47","Type":"ContainerStarted","Data":"77df98855940a98a24078fc182556c2875bbc4770bdcac8637d01eb654f3aea4"}
Nov 03 12:23:47.509263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:47.508902    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:47.614474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:47.614425    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9" podStartSLOduration=2.61441355 podStartE2EDuration="2.61441355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:47.613462468 +0000 UTC m=+1969.185046384" watchObservedRunningTime="2026-11-03 12:23:47.61441355 +0000 UTC m=+1969.185997461"
Nov 03 12:23:51.081443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:51.081413    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"]
Nov 03 12:23:51.081999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:51.081620    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" containerID="cri-o://abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6" gracePeriod=30
Nov 03 12:23:53.045629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:53.045599    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"]
Nov 03 12:23:53.146220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:53.146153    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"]
Nov 03 12:23:54.152158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.152116    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"]
Nov 03 12:23:54.156319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.156298    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.206206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.206181    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-583bcf\""
Nov 03 12:23:54.206582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.206564    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-583bcf-dockercfg-l7ctf\""
Nov 03 12:23:54.210038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.210013    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"]
Nov 03 12:23:54.290403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.290376    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.290550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.290433    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx97x\" (UniqueName: \"kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x\") pod \"payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.391843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.391805    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx97x\" (UniqueName: \"kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x\") pod \"payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.392099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.392077    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.392723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.392689    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.458818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.458463    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx97x\" (UniqueName: \"kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x\") pod \"payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.470466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.470341    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:54.605883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.605854    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"]
Nov 03 12:23:54.610200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.610181    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:54.664038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.664008    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"]
Nov 03 12:23:54.797797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.797742    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljzf\" (UniqueName: \"kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:54.798010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.797966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:54.837900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.837874    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"]
Nov 03 12:23:54.839151 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:54.839123    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edff79f_390a_4517_a24e_f3494430573c.slice/crio-5fcabf6f7931eef33223d275ca66dc04f8ee72318075d571af63214182c8d60f WatchSource:0}: Error finding container 5fcabf6f7931eef33223d275ca66dc04f8ee72318075d571af63214182c8d60f: Status 404 returned error can't find the container with id 5fcabf6f7931eef33223d275ca66dc04f8ee72318075d571af63214182c8d60f
Nov 03 12:23:54.899072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.899046    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljzf\" (UniqueName: \"kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:54.899212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.899150    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:54.899845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.899825    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:54.952332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:54.952304    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljzf\" (UniqueName: \"kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf\") pod \"payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:55.226116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:55.226085    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:55.591330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:55.591225    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2" event={"ID":"3edff79f-390a-4517-a24e-f3494430573c","Type":"ContainerStarted","Data":"c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad"}
Nov 03 12:23:55.591330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:55.591329    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2" event={"ID":"3edff79f-390a-4517-a24e-f3494430573c","Type":"ContainerStarted","Data":"5fcabf6f7931eef33223d275ca66dc04f8ee72318075d571af63214182c8d60f"}
Nov 03 12:23:55.591569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:55.591461    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:23:55.602695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:55.602666    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"]
Nov 03 12:23:55.603200 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:23:55.603177    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d79af3_646b_4732_bff3_ff13a607ac9b.slice/crio-bee6799391dacd98163c19edf693d0b072532f132fc80e99896f9771f24fa127 WatchSource:0}: Error finding container bee6799391dacd98163c19edf693d0b072532f132fc80e99896f9771f24fa127: Status 404 returned error can't find the container with id bee6799391dacd98163c19edf693d0b072532f132fc80e99896f9771f24fa127
Nov 03 12:23:55.698703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:55.698647    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2" podStartSLOduration=1.698628919 podStartE2EDuration="1.698628919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:55.696209345 +0000 UTC m=+1977.267793259" watchObservedRunningTime="2026-11-03 12:23:55.698628919 +0000 UTC m=+1977.270212835"
Nov 03 12:23:56.099299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.099274    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 12:23:56.099426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.099329    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 12:23:56.346641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.346579    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:23:56.418730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.418695    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.418888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.418807    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.418888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.418836    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.418894    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.418979    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:56.419131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419022    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419127    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419128    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:56.419198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419159    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419230    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419267    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy\") pod \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\" (UID: \"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6\") "
Nov 03 12:23:56.419370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419336    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:56.419509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419438    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:56.419834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.419792    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:56.420037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.420019    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.420120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.420045    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.420120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.420061    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.420120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.420076    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.420120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.420105    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.421331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.421300    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:56.421427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.421408    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:56.421655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.421640    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:56.421856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.421843    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz" pod "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" (UID: "f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6"). InnerVolumeSpecName "kube-api-access-rvxkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:56.521701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.521666    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.521701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.521699    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvxkz\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-kube-api-access-rvxkz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.521890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.521716    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.521890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.521731    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:56.599328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.599243    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r" event={"ID":"69d79af3-646b-4732-bff3-ff13a607ac9b","Type":"ContainerStarted","Data":"e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7"}
Nov 03 12:23:56.599328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.599304    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r" event={"ID":"69d79af3-646b-4732-bff3-ff13a607ac9b","Type":"ContainerStarted","Data":"bee6799391dacd98163c19edf693d0b072532f132fc80e99896f9771f24fa127"}
Nov 03 12:23:56.599525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.599446    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:23:56.601051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.601031    4546 generic.go:356] "Generic (PLEG): container finished" podID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerID="abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6" exitCode=0
Nov 03 12:23:56.601193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.601083    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"
Nov 03 12:23:56.601193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.601111    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" event={"ID":"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6","Type":"ContainerDied","Data":"abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6"}
Nov 03 12:23:56.601193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.601162    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h" event={"ID":"f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6","Type":"ContainerDied","Data":"4a40c93d21a9515da65eec0a473ba5fe92167bf030d0fecb06e96e267fead0c0"}
Nov 03 12:23:56.601193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.601185    4546 scope.go:122] "RemoveContainer" containerID="abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6"
Nov 03 12:23:56.612218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.612199    4546 scope.go:122] "RemoveContainer" containerID="a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"
Nov 03 12:23:56.622945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.622923    4546 scope.go:122] "RemoveContainer" containerID="abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6"
Nov 03 12:23:56.623582 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:56.623454    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6\": container with ID starting with abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6 not found: ID does not exist" containerID="abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6"
Nov 03 12:23:56.623582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.623493    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6"} err="failed to get container status \"abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6\": rpc error: code = NotFound desc = could not find container \"abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6\": container with ID starting with abd51020d726c30a95f0ba890eea495747ae61536e2d9b8605023731c68f7bf6 not found: ID does not exist"
Nov 03 12:23:56.623582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.623518    4546 scope.go:122] "RemoveContainer" containerID="a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"
Nov 03 12:23:56.624126 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:56.624102    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1\": container with ID starting with a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1 not found: ID does not exist" containerID="a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"
Nov 03 12:23:56.624202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.624132    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1"} err="failed to get container status \"a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1\": rpc error: code = NotFound desc = could not find container \"a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1\": container with ID starting with a380526dcc429c9ccb092cd7a9c53ea3308fe6537947fe67f48a831bd1e6f5b1 not found: ID does not exist"
Nov 03 12:23:56.706542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.706490    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r" podStartSLOduration=2.706477608 podStartE2EDuration="2.706477608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:23:56.70400674 +0000 UTC m=+1978.275590653" watchObservedRunningTime="2026-11-03 12:23:56.706477608 +0000 UTC m=+1978.278061520"
Nov 03 12:23:56.809718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.809688    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"]
Nov 03 12:23:56.812800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:56.812775    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e151c-openshift-default-58c67cf999-sdp9h"]
Nov 03 12:23:57.512545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:57.512517    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:23:58.045899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.045859    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr" podUID="c052d6b6-c238-4cb4-9bd9-b839179ef2b2" containerName="payload-processing" containerID="cri-o://e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e" gracePeriod=25
Nov 03 12:23:58.146832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.146768    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l" podUID="3e384abe-3c71-414e-adf5-c4668a6c7b98" containerName="payload-pre-processing" containerID="cri-o://8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f" gracePeriod=25
Nov 03 12:23:58.344341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.344313    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:23:58.452990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.452935    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8\" (UniqueName: \"kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8\") pod \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") "
Nov 03 12:23:58.453149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.453004    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume\") pod \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\" (UID: \"c052d6b6-c238-4cb4-9bd9-b839179ef2b2\") "
Nov 03 12:23:58.453836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.453799    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume" pod "c052d6b6-c238-4cb4-9bd9-b839179ef2b2" (UID: "c052d6b6-c238-4cb4-9bd9-b839179ef2b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:58.454508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.454480    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:58.455643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.455620    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8" pod "c052d6b6-c238-4cb4-9bd9-b839179ef2b2" (UID: "c052d6b6-c238-4cb4-9bd9-b839179ef2b2"). InnerVolumeSpecName "kube-api-access-m6hc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:58.461421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.461402    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:23:58.518488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.518466    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:23:58.555342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.555257    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w\" (UniqueName: \"kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w\") pod \"3e384abe-3c71-414e-adf5-c4668a6c7b98\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") "
Nov 03 12:23:58.555568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.555549    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume\") pod \"3e384abe-3c71-414e-adf5-c4668a6c7b98\" (UID: \"3e384abe-3c71-414e-adf5-c4668a6c7b98\") "
Nov 03 12:23:58.556029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.556000    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume" pod "3e384abe-3c71-414e-adf5-c4668a6c7b98" (UID: "3e384abe-3c71-414e-adf5-c4668a6c7b98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:58.556137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.556026    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hc8\" (UniqueName: \"kubernetes.io/projected/c052d6b6-c238-4cb4-9bd9-b839179ef2b2-kube-api-access-m6hc8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:58.557778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.557737    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w" pod "3e384abe-3c71-414e-adf5-c4668a6c7b98" (UID: "3e384abe-3c71-414e-adf5-c4668a6c7b98"). InnerVolumeSpecName "kube-api-access-4244w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:58.577978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.577947    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" path="/var/lib/kubelet/pods/f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6/volumes"
Nov 03 12:23:58.619432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.619403    4546 generic.go:356] "Generic (PLEG): container finished" podID="c052d6b6-c238-4cb4-9bd9-b839179ef2b2" containerID="e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e" exitCode=0
Nov 03 12:23:58.619590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.619453    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"
Nov 03 12:23:58.619590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.619476    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr" event={"ID":"c052d6b6-c238-4cb4-9bd9-b839179ef2b2","Type":"ContainerDied","Data":"e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e"}
Nov 03 12:23:58.619590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.619570    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr" event={"ID":"c052d6b6-c238-4cb4-9bd9-b839179ef2b2","Type":"ContainerDied","Data":"bf52839711a5eb926798d62cabfe39e4268e0a4ce347962e4fbfc6bdfff7aa1a"}
Nov 03 12:23:58.619782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.619598    4546 scope.go:122] "RemoveContainer" containerID="e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e"
Nov 03 12:23:58.622721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.622541    4546 generic.go:356] "Generic (PLEG): container finished" podID="3e384abe-3c71-414e-adf5-c4668a6c7b98" containerID="8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f" exitCode=0
Nov 03 12:23:58.622721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.622580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l" event={"ID":"3e384abe-3c71-414e-adf5-c4668a6c7b98","Type":"ContainerDied","Data":"8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f"}
Nov 03 12:23:58.622721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.622620    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l" event={"ID":"3e384abe-3c71-414e-adf5-c4668a6c7b98","Type":"ContainerDied","Data":"48fcc3606ed2c91433dfbeeaa06ce50ad2e6681b7c56a24cfbd2b6f48a78d1cf"}
Nov 03 12:23:58.622721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.622687    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"
Nov 03 12:23:58.631847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.630327    4546 scope.go:122] "RemoveContainer" containerID="e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e"
Nov 03 12:23:58.631847 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:58.630607    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e\": container with ID starting with e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e not found: ID does not exist" containerID="e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e"
Nov 03 12:23:58.631847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.630640    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e"} err="failed to get container status \"e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e\": rpc error: code = NotFound desc = could not find container \"e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e\": container with ID starting with e1809cf92e1d5881bb32a3a3256ef0ae72bd0f20ab3d1d41f6f19c7b0e54751e not found: ID does not exist"
Nov 03 12:23:58.631847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.630657    4546 scope.go:122] "RemoveContainer" containerID="8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f"
Nov 03 12:23:58.639949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.639925    4546 scope.go:122] "RemoveContainer" containerID="8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f"
Nov 03 12:23:58.640322 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:23:58.640295    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f\": container with ID starting with 8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f not found: ID does not exist" containerID="8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f"
Nov 03 12:23:58.640444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.640330    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f"} err="failed to get container status \"8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f\": rpc error: code = NotFound desc = could not find container \"8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f\": container with ID starting with 8ee4446322e949e6856c92440bb1195c63e0dc1ae6acabc758893add2813540f not found: ID does not exist"
Nov 03 12:23:58.656908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.656889    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e384abe-3c71-414e-adf5-c4668a6c7b98-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:58.656908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.656907    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4244w\" (UniqueName: \"kubernetes.io/projected/3e384abe-3c71-414e-adf5-c4668a6c7b98-kube-api-access-4244w\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:58.831586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.831515    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"]
Nov 03 12:23:58.834908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.834882    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-059838-55d95fb7c9-br62l"]
Nov 03 12:23:58.935491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.935456    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"]
Nov 03 12:23:58.939241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:23:58.939213    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-059838-d64f6569c-pztpr"]
Nov 03 12:24:00.063359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:00.063330    4546 scope.go:122] "RemoveContainer" containerID="a2b83dea795c20134776f0d48abbd3736be0a9ad84cad47a0b01d9a0547fac73"
Nov 03 12:24:00.073285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:00.073263    4546 scope.go:122] "RemoveContainer" containerID="247013ab9eb3b42207a53409817ee489154487f7f1d7662044175f2c257b1601"
Nov 03 12:24:00.577875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:00.577834    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e384abe-3c71-414e-adf5-c4668a6c7b98" path="/var/lib/kubelet/pods/3e384abe-3c71-414e-adf5-c4668a6c7b98/volumes"
Nov 03 12:24:00.578524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:00.578491    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c052d6b6-c238-4cb4-9bd9-b839179ef2b2" path="/var/lib/kubelet/pods/c052d6b6-c238-4cb4-9bd9-b839179ef2b2/volumes"
Nov 03 12:24:06.603918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:06.603881    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:24:07.611085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:07.611054    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:24:07.654295 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:07.654261    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:24:19.453590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.453537    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"]
Nov 03 12:24:19.454261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454245    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy"
Nov 03 12:24:19.454307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454264    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy"
Nov 03 12:24:19.454307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454288    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy"
Nov 03 12:24:19.454307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454293    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy"
Nov 03 12:24:19.454307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454306    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3e384abe-3c71-414e-adf5-c4668a6c7b98" containerName="payload-pre-processing"
Nov 03 12:24:19.454444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454312    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e384abe-3c71-414e-adf5-c4668a6c7b98" containerName="payload-pre-processing"
Nov 03 12:24:19.454444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454323    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c052d6b6-c238-4cb4-9bd9-b839179ef2b2" containerName="payload-processing"
Nov 03 12:24:19.454444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454330    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c052d6b6-c238-4cb4-9bd9-b839179ef2b2" containerName="payload-processing"
Nov 03 12:24:19.454444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454431    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy"
Nov 03 12:24:19.454444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454440    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1caa903-f2f8-4c8f-a8e2-7550ec5f16a6" containerName="istio-proxy"
Nov 03 12:24:19.454599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454447    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e384abe-3c71-414e-adf5-c4668a6c7b98" containerName="payload-pre-processing"
Nov 03 12:24:19.454599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.454455    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c052d6b6-c238-4cb4-9bd9-b839179ef2b2" containerName="payload-processing"
Nov 03 12:24:19.458001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.457976    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.507617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.507585    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:24:19.553904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.553868    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:24:19.553904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.553868    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:24:19.554167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.553943    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:24:19.559247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.559213    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"]
Nov 03 12:24:19.602594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.602555    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.602777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.602674    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rs7g\" (UniqueName: \"kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.602777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.602768    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.703549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.703516    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rs7g\" (UniqueName: \"kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.703850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.703617    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.703950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.703929    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.704560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.704538    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.706351 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.706333    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.756940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.756906    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rs7g\" (UniqueName: \"kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g\") pod \"maas-api-revoke-sub-a26d042c-cn7t7\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:19.771643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:19.771619    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:20.146505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:20.146482    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"]
Nov 03 12:24:20.147596 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:24:20.147564    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcd6531_55b2_42b5_917b_43f5bc496c33.slice/crio-d2f7d93214007f831f316256fecbd177a47bd2249edb3a34ed0317e7c06afb9e WatchSource:0}: Error finding container d2f7d93214007f831f316256fecbd177a47bd2249edb3a34ed0317e7c06afb9e: Status 404 returned error can't find the container with id d2f7d93214007f831f316256fecbd177a47bd2249edb3a34ed0317e7c06afb9e
Nov 03 12:24:20.762003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:20.761969    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7" event={"ID":"fdcd6531-55b2-42b5-917b-43f5bc496c33","Type":"ContainerStarted","Data":"366532d9b50871ac5307f803f66442e1ddeed14a6427a795cc7540df74cba87f"}
Nov 03 12:24:20.762364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:20.762020    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7" event={"ID":"fdcd6531-55b2-42b5-917b-43f5bc496c33","Type":"ContainerStarted","Data":"d2f7d93214007f831f316256fecbd177a47bd2249edb3a34ed0317e7c06afb9e"}
Nov 03 12:24:20.870688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:20.870633    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7" podStartSLOduration=1.870619385 podStartE2EDuration="1.870619385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:24:20.867064129 +0000 UTC m=+2002.438648041" watchObservedRunningTime="2026-11-03 12:24:20.870619385 +0000 UTC m=+2002.442203301"
Nov 03 12:24:21.769641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:21.769609    4546 generic.go:356] "Generic (PLEG): container finished" podID="fdcd6531-55b2-42b5-917b-43f5bc496c33" containerID="366532d9b50871ac5307f803f66442e1ddeed14a6427a795cc7540df74cba87f" exitCode=0
Nov 03 12:24:21.770114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:21.769692    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7" event={"ID":"fdcd6531-55b2-42b5-917b-43f5bc496c33","Type":"ContainerDied","Data":"366532d9b50871ac5307f803f66442e1ddeed14a6427a795cc7540df74cba87f"}
Nov 03 12:24:22.937736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.937707    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:22.939829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.939811    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g\" (UniqueName: \"kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g\") pod \"fdcd6531-55b2-42b5-917b-43f5bc496c33\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") "
Nov 03 12:24:22.939905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.939862    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca\") pod \"fdcd6531-55b2-42b5-917b-43f5bc496c33\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") "
Nov 03 12:24:22.939905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.939891    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca\") pod \"fdcd6531-55b2-42b5-917b-43f5bc496c33\" (UID: \"fdcd6531-55b2-42b5-917b-43f5bc496c33\") "
Nov 03 12:24:22.940238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.940211    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca" pod "fdcd6531-55b2-42b5-917b-43f5bc496c33" (UID: "fdcd6531-55b2-42b5-917b-43f5bc496c33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:22.942053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.942017    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g" pod "fdcd6531-55b2-42b5-917b-43f5bc496c33" (UID: "fdcd6531-55b2-42b5-917b-43f5bc496c33"). InnerVolumeSpecName "kube-api-access-7rs7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:22.942144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:22.942103    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca" pod "fdcd6531-55b2-42b5-917b-43f5bc496c33" (UID: "fdcd6531-55b2-42b5-917b-43f5bc496c33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:23.041059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:23.040983    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rs7g\" (UniqueName: \"kubernetes.io/projected/fdcd6531-55b2-42b5-917b-43f5bc496c33-kube-api-access-7rs7g\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:23.041059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:23.041014    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:23.041059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:23.041030    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcd6531-55b2-42b5-917b-43f5bc496c33-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:23.792963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:23.792923    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"
Nov 03 12:24:23.792963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:23.792934    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7" event={"ID":"fdcd6531-55b2-42b5-917b-43f5bc496c33","Type":"ContainerDied","Data":"d2f7d93214007f831f316256fecbd177a47bd2249edb3a34ed0317e7c06afb9e"}
Nov 03 12:24:23.793191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:23.792996    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f7d93214007f831f316256fecbd177a47bd2249edb3a34ed0317e7c06afb9e"
Nov 03 12:24:29.074321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.074284    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"]
Nov 03 12:24:29.074873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.074524    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5" podUID="91625b58-be1c-48af-a2d9-1b7d143e7e4c" containerName="payload-processing" containerID="cri-o://d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6" gracePeriod=30
Nov 03 12:24:29.138442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.138399    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"]
Nov 03 12:24:29.138945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.138918    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9" podUID="f014f23e-d756-4dbc-9dc7-65928b1cc013" containerName="payload-pre-processing" containerID="cri-o://ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579" gracePeriod=30
Nov 03 12:24:29.493066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.493041    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:24:29.496266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.496245    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:24:29.653827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.652736    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume\") pod \"f014f23e-d756-4dbc-9dc7-65928b1cc013\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") "
Nov 03 12:24:29.653827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.652901    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume\") pod \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") "
Nov 03 12:24:29.653827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.652976    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f\" (UniqueName: \"kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f\") pod \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\" (UID: \"91625b58-be1c-48af-a2d9-1b7d143e7e4c\") "
Nov 03 12:24:29.653827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.653099    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92\" (UniqueName: \"kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92\") pod \"f014f23e-d756-4dbc-9dc7-65928b1cc013\" (UID: \"f014f23e-d756-4dbc-9dc7-65928b1cc013\") "
Nov 03 12:24:29.654296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.654233    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume" pod "91625b58-be1c-48af-a2d9-1b7d143e7e4c" (UID: "91625b58-be1c-48af-a2d9-1b7d143e7e4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:29.654601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.654570    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume" pod "f014f23e-d756-4dbc-9dc7-65928b1cc013" (UID: "f014f23e-d756-4dbc-9dc7-65928b1cc013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:29.656398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.656357    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92" pod "f014f23e-d756-4dbc-9dc7-65928b1cc013" (UID: "f014f23e-d756-4dbc-9dc7-65928b1cc013"). InnerVolumeSpecName "kube-api-access-4cs92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:29.658349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.658300    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f" pod "91625b58-be1c-48af-a2d9-1b7d143e7e4c" (UID: "91625b58-be1c-48af-a2d9-1b7d143e7e4c"). InnerVolumeSpecName "kube-api-access-hjd2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:29.754664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.754627    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cs92\" (UniqueName: \"kubernetes.io/projected/f014f23e-d756-4dbc-9dc7-65928b1cc013-kube-api-access-4cs92\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.754664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.754665    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f014f23e-d756-4dbc-9dc7-65928b1cc013-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.754899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.754681    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91625b58-be1c-48af-a2d9-1b7d143e7e4c-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.754899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.754696    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjd2f\" (UniqueName: \"kubernetes.io/projected/91625b58-be1c-48af-a2d9-1b7d143e7e4c-kube-api-access-hjd2f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.839507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.839472    4546 generic.go:356] "Generic (PLEG): container finished" podID="f014f23e-d756-4dbc-9dc7-65928b1cc013" containerID="ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579" exitCode=0
Nov 03 12:24:29.839677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.839515    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"
Nov 03 12:24:29.839677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.839549    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9" event={"ID":"f014f23e-d756-4dbc-9dc7-65928b1cc013","Type":"ContainerDied","Data":"ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579"}
Nov 03 12:24:29.839677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.839603    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9" event={"ID":"f014f23e-d756-4dbc-9dc7-65928b1cc013","Type":"ContainerDied","Data":"bd340dc20015347d030d16fc4e8c60faf9f2a6abb2bace8f56806a81fb79c5d2"}
Nov 03 12:24:29.839677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.839631    4546 scope.go:122] "RemoveContainer" containerID="ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579"
Nov 03 12:24:29.841195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.841173    4546 generic.go:356] "Generic (PLEG): container finished" podID="91625b58-be1c-48af-a2d9-1b7d143e7e4c" containerID="d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6" exitCode=0
Nov 03 12:24:29.841316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.841212    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"
Nov 03 12:24:29.841316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.841228    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5" event={"ID":"91625b58-be1c-48af-a2d9-1b7d143e7e4c","Type":"ContainerDied","Data":"d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6"}
Nov 03 12:24:29.841316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.841259    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5" event={"ID":"91625b58-be1c-48af-a2d9-1b7d143e7e4c","Type":"ContainerDied","Data":"e95892c46242de2e269daf5e548f64255e0fed8e89b65ddda529312c5d98a424"}
Nov 03 12:24:29.850599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.850402    4546 scope.go:122] "RemoveContainer" containerID="ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579"
Nov 03 12:24:29.850785 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:29.850724    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579\": container with ID starting with ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579 not found: ID does not exist" containerID="ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579"
Nov 03 12:24:29.850870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.850800    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579"} err="failed to get container status \"ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579\": rpc error: code = NotFound desc = could not find container \"ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579\": container with ID starting with ae5f0d39a5f9c846e723bb7b5f72752466f8b383e1b3e55d91b59d0790ce5579 not found: ID does not exist"
Nov 03 12:24:29.850870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.850824    4546 scope.go:122] "RemoveContainer" containerID="d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6"
Nov 03 12:24:29.859851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.859831    4546 scope.go:122] "RemoveContainer" containerID="d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6"
Nov 03 12:24:29.860111 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:29.860091    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6\": container with ID starting with d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6 not found: ID does not exist" containerID="d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6"
Nov 03 12:24:29.860171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.860118    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6"} err="failed to get container status \"d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6\": rpc error: code = NotFound desc = could not find container \"d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6\": container with ID starting with d744f587a89b5fe6fe4d4d9cefda92fb6983c0b1059748d4dcb57ba28ffa35c6 not found: ID does not exist"
Nov 03 12:24:29.991336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.991299    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"]
Nov 03 12:24:29.994632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:29.994607    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-deb666-7d67dc96d8-bg2j5"]
Nov 03 12:24:30.096361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:30.096333    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"]
Nov 03 12:24:30.099891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:30.099861    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-deb666-5ff9d87b55-d4sr9"]
Nov 03 12:24:30.574545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:30.574514    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91625b58-be1c-48af-a2d9-1b7d143e7e4c" path="/var/lib/kubelet/pods/91625b58-be1c-48af-a2d9-1b7d143e7e4c/volumes"
Nov 03 12:24:30.574923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:30.574910    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f014f23e-d756-4dbc-9dc7-65928b1cc013" path="/var/lib/kubelet/pods/f014f23e-d756-4dbc-9dc7-65928b1cc013/volumes"
Nov 03 12:24:32.643477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:32.625484    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"]
Nov 03 12:24:32.643477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:32.625741    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy" containerID="cri-o://1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7" gracePeriod=30
Nov 03 12:24:37.657316 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:37.657267    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:24:37.894881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.894859    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:24:37.913574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.913513    4546 generic.go:356] "Generic (PLEG): container finished" podID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerID="1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7" exitCode=0
Nov 03 12:24:37.913574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.913563    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"
Nov 03 12:24:37.913785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.913600    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" event={"ID":"d52bbfaa-1be7-40e7-9caa-997651d7155c","Type":"ContainerDied","Data":"1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7"}
Nov 03 12:24:37.913785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.913655    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv" event={"ID":"d52bbfaa-1be7-40e7-9caa-997651d7155c","Type":"ContainerDied","Data":"4f7079f3bca5f68bba34fcb1af9b56499b8e37362afdba41570df2eb46f2d0f5"}
Nov 03 12:24:37.913785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.913677    4546 scope.go:122] "RemoveContainer" containerID="1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7"
Nov 03 12:24:37.923036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.923013    4546 scope.go:122] "RemoveContainer" containerID="07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"
Nov 03 12:24:37.931719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.931686    4546 scope.go:122] "RemoveContainer" containerID="1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7"
Nov 03 12:24:37.932033 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:37.932012    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7\": container with ID starting with 1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7 not found: ID does not exist" containerID="1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7"
Nov 03 12:24:37.932109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.932040    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7"} err="failed to get container status \"1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7\": rpc error: code = NotFound desc = could not find container \"1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7\": container with ID starting with 1ca91660ade39460d2fe64258b54c6b993c82382faef80aea6601b9dcd57c3c7 not found: ID does not exist"
Nov 03 12:24:37.932109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.932057    4546 scope.go:122] "RemoveContainer" containerID="07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"
Nov 03 12:24:37.932303 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:37.932281    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad\": container with ID starting with 07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad not found: ID does not exist" containerID="07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"
Nov 03 12:24:37.932345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.932313    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad"} err="failed to get container status \"07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad\": rpc error: code = NotFound desc = could not find container \"07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad\": container with ID starting with 07e783e019aa937a1364851dce50aedcb85ea4ed7d83251155066d6354961aad not found: ID does not exist"
Nov 03 12:24:37.986234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986205    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986247    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986392    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986431    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986465    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986488    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986540    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986610    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986614    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:37.986737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986644    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token\") pod \"d52bbfaa-1be7-40e7-9caa-997651d7155c\" (UID: \"d52bbfaa-1be7-40e7-9caa-997651d7155c\") "
Nov 03 12:24:37.986930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986776    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:37.986930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986800    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:37.986930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.986858    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:37.987258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.987165    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:37.987323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.987273    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:37.987323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.987293    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52bbfaa-1be7-40e7-9caa-997651d7155c-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:37.987323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.987307    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:37.987323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.987321    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:37.987542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.987335    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:37.988923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.988889    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "kube-api-access-4mchn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:37.989143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.989107    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:37.990696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.990670    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:37.990815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:37.990707    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token" pod "d52bbfaa-1be7-40e7-9caa-997651d7155c" (UID: "d52bbfaa-1be7-40e7-9caa-997651d7155c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:38.088459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.088429    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mchn\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-kube-api-access-4mchn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:38.088459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.088455    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:38.088459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.088464    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:38.088684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.088473    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52bbfaa-1be7-40e7-9caa-997651d7155c-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:38.327790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.327661    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"]
Nov 03 12:24:38.331656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.331619    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-deb666-openshift-default-775b79f64f-w25cv"]
Nov 03 12:24:38.575240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.575162    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" path="/var/lib/kubelet/pods/d52bbfaa-1be7-40e7-9caa-997651d7155c/volumes"
Nov 03 12:24:38.923630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.923550    4546 generic.go:356] "Generic (PLEG): container finished" podID="b90b2734-457b-454d-b91d-d10560505147" containerID="16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060" exitCode=0
Nov 03 12:24:38.923630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.923619    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" event={"ID":"b90b2734-457b-454d-b91d-d10560505147","Type":"ContainerDied","Data":"16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"}
Nov 03 12:24:38.924222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.923998    4546 scope.go:122] "RemoveContainer" containerID="16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"
Nov 03 12:24:38.925386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.925353    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:38.925506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.925434    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:38.925506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:38.925481    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:39.939037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:39.938998    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" event={"ID":"b90b2734-457b-454d-b91d-d10560505147","Type":"ContainerStarted","Data":"452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2"}
Nov 03 12:24:40.058583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:40.058552    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:24:40.058795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:40.058703    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 12:24:40.058857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:40.058786    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 12:24:41.046683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.046642    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"]
Nov 03 12:24:41.047143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.046898    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2" podUID="3edff79f-390a-4517-a24e-f3494430573c" containerName="payload-processing" containerID="cri-o://c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad" gracePeriod=30
Nov 03 12:24:41.058569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.058542    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 12:24:41.058685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.058606    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 12:24:41.093688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.093661    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"]
Nov 03 12:24:41.093922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.093899    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r" podUID="69d79af3-646b-4732-bff3-ff13a607ac9b" containerName="payload-pre-processing" containerID="cri-o://e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7" gracePeriod=30
Nov 03 12:24:41.445312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.445280    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:24:41.448696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.448673    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:24:41.452256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452235    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume\") pod \"3edff79f-390a-4517-a24e-f3494430573c\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") "
Nov 03 12:24:41.452395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452284    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume\") pod \"69d79af3-646b-4732-bff3-ff13a607ac9b\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") "
Nov 03 12:24:41.452395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452319    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf\" (UniqueName: \"kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf\") pod \"69d79af3-646b-4732-bff3-ff13a607ac9b\" (UID: \"69d79af3-646b-4732-bff3-ff13a607ac9b\") "
Nov 03 12:24:41.452395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452387    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x\" (UniqueName: \"kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x\") pod \"3edff79f-390a-4517-a24e-f3494430573c\" (UID: \"3edff79f-390a-4517-a24e-f3494430573c\") "
Nov 03 12:24:41.452739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452710    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume" pod "3edff79f-390a-4517-a24e-f3494430573c" (UID: "3edff79f-390a-4517-a24e-f3494430573c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:41.452860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452738    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume" pod "69d79af3-646b-4732-bff3-ff13a607ac9b" (UID: "69d79af3-646b-4732-bff3-ff13a607ac9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:41.452931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452904    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edff79f-390a-4517-a24e-f3494430573c-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.452984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.452936    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d79af3-646b-4732-bff3-ff13a607ac9b-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.454712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.454688    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf" pod "69d79af3-646b-4732-bff3-ff13a607ac9b" (UID: "69d79af3-646b-4732-bff3-ff13a607ac9b"). InnerVolumeSpecName "kube-api-access-lljzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:41.454840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.454743    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x" pod "3edff79f-390a-4517-a24e-f3494430573c" (UID: "3edff79f-390a-4517-a24e-f3494430573c"). InnerVolumeSpecName "kube-api-access-xx97x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:41.590842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.583238    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lljzf\" (UniqueName: \"kubernetes.io/projected/69d79af3-646b-4732-bff3-ff13a607ac9b-kube-api-access-lljzf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.590842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.583272    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx97x\" (UniqueName: \"kubernetes.io/projected/3edff79f-390a-4517-a24e-f3494430573c-kube-api-access-xx97x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:41.955134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.955102    4546 generic.go:356] "Generic (PLEG): container finished" podID="69d79af3-646b-4732-bff3-ff13a607ac9b" containerID="e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7" exitCode=0
Nov 03 12:24:41.955322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.955151    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"
Nov 03 12:24:41.955322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.955188    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r" event={"ID":"69d79af3-646b-4732-bff3-ff13a607ac9b","Type":"ContainerDied","Data":"e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7"}
Nov 03 12:24:41.955322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.955241    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r" event={"ID":"69d79af3-646b-4732-bff3-ff13a607ac9b","Type":"ContainerDied","Data":"bee6799391dacd98163c19edf693d0b072532f132fc80e99896f9771f24fa127"}
Nov 03 12:24:41.955322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.955264    4546 scope.go:122] "RemoveContainer" containerID="e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7"
Nov 03 12:24:41.956663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.956641    4546 generic.go:356] "Generic (PLEG): container finished" podID="3edff79f-390a-4517-a24e-f3494430573c" containerID="c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad" exitCode=0
Nov 03 12:24:41.956824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.956661    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2" event={"ID":"3edff79f-390a-4517-a24e-f3494430573c","Type":"ContainerDied","Data":"c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad"}
Nov 03 12:24:41.956824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.956691    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"
Nov 03 12:24:41.956824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.956714    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2" event={"ID":"3edff79f-390a-4517-a24e-f3494430573c","Type":"ContainerDied","Data":"5fcabf6f7931eef33223d275ca66dc04f8ee72318075d571af63214182c8d60f"}
Nov 03 12:24:41.966363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.966291    4546 scope.go:122] "RemoveContainer" containerID="e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7"
Nov 03 12:24:41.966624 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:41.966599    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7\": container with ID starting with e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7 not found: ID does not exist" containerID="e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7"
Nov 03 12:24:41.966728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.966626    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7"} err="failed to get container status \"e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7\": rpc error: code = NotFound desc = could not find container \"e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7\": container with ID starting with e36aee161917393f4d197639c32c9a154b9ca133b5e94e9237f9e7992dc30bf7 not found: ID does not exist"
Nov 03 12:24:41.966728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.966645    4546 scope.go:122] "RemoveContainer" containerID="c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad"
Nov 03 12:24:41.976208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.976190    4546 scope.go:122] "RemoveContainer" containerID="c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad"
Nov 03 12:24:41.976467 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:24:41.976447    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad\": container with ID starting with c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad not found: ID does not exist" containerID="c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad"
Nov 03 12:24:41.976532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:41.976477    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad"} err="failed to get container status \"c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad\": rpc error: code = NotFound desc = could not find container \"c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad\": container with ID starting with c8d463fad26c1ad90e11df952a9f381ef3d270fb0c14898969963eb65573d0ad not found: ID does not exist"
Nov 03 12:24:42.058884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.058848    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 12:24:42.059248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.058904    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 12:24:42.129781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.119814    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"]
Nov 03 12:24:42.133987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.133953    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-583bcf-856f6456c-ljz8r"]
Nov 03 12:24:42.235900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.235871    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"]
Nov 03 12:24:42.238896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.238861    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-583bcf-5c7c6d759d-nwlx2"]
Nov 03 12:24:42.573924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.573828    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edff79f-390a-4517-a24e-f3494430573c" path="/var/lib/kubelet/pods/3edff79f-390a-4517-a24e-f3494430573c/volumes"
Nov 03 12:24:42.574332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:42.574310    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d79af3-646b-4732-bff3-ff13a607ac9b" path="/var/lib/kubelet/pods/69d79af3-646b-4732-bff3-ff13a607ac9b/volumes"
Nov 03 12:24:43.059253 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:43.059217    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 12:24:43.059669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:43.059294    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 12:24:44.058583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:44.058550    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 12:24:44.058814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:44.058600    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 12:24:45.058965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:45.058928    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 12:24:45.059388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:45.058995    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 12:24:46.063070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:46.063040    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:24:46.063456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:46.063307    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:24:46.063950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:46.063934    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:24:46.798501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:46.798467    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"]
Nov 03 12:24:46.798729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:46.798706    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy" containerID="cri-o://d30d4dd0a9ded501937b541b9d522827ca789e8014ef8d4e1c77df86e8db513a" gracePeriod=30
Nov 03 12:24:48.798468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:48.798429    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"]
Nov 03 12:24:48.799358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:48.799321    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" containerID="cri-o://802b3dedab59b0e7deed1c840878c01d36c8c35de62be2ed12a79c9d53db2a37" gracePeriod=30
Nov 03 12:24:50.083602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.083567    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"]
Nov 03 12:24:50.085169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.085141    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f014f23e-d756-4dbc-9dc7-65928b1cc013" containerName="payload-pre-processing"
Nov 03 12:24:50.086836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.085332    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f014f23e-d756-4dbc-9dc7-65928b1cc013" containerName="payload-pre-processing"
Nov 03 12:24:50.086976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.086961    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3edff79f-390a-4517-a24e-f3494430573c" containerName="payload-processing"
Nov 03 12:24:50.087070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.087058    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edff79f-390a-4517-a24e-f3494430573c" containerName="payload-processing"
Nov 03 12:24:50.087171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.087159    4546 cpu_manager.go:414] "containerMap: removing container" podUID="91625b58-be1c-48af-a2d9-1b7d143e7e4c" containerName="payload-processing"
Nov 03 12:24:50.087250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.087238    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91625b58-be1c-48af-a2d9-1b7d143e7e4c" containerName="payload-processing"
Nov 03 12:24:50.087338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.087329    4546 cpu_manager.go:414] "containerMap: removing container" podUID="69d79af3-646b-4732-bff3-ff13a607ac9b" containerName="payload-pre-processing"
Nov 03 12:24:50.087413 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.087402    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d79af3-646b-4732-bff3-ff13a607ac9b" containerName="payload-pre-processing"
Nov 03 12:24:50.090637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.090617    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy"
Nov 03 12:24:50.090815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.090800    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy"
Nov 03 12:24:50.090938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.090924    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy"
Nov 03 12:24:50.091031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.091019    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy"
Nov 03 12:24:50.091119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.091108    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fdcd6531-55b2-42b5-917b-43f5bc496c33" containerName="revoke-keys"
Nov 03 12:24:50.091196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.091184    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcd6531-55b2-42b5-917b-43f5bc496c33" containerName="revoke-keys"
Nov 03 12:24:50.096126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.091446    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy"
Nov 03 12:24:50.096268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.096252    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcd6531-55b2-42b5-917b-43f5bc496c33" containerName="revoke-keys"
Nov 03 12:24:50.096382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.096370    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f014f23e-d756-4dbc-9dc7-65928b1cc013" containerName="payload-pre-processing"
Nov 03 12:24:50.096458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.096448    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edff79f-390a-4517-a24e-f3494430573c" containerName="payload-processing"
Nov 03 12:24:50.096530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.096521    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="91625b58-be1c-48af-a2d9-1b7d143e7e4c" containerName="payload-processing"
Nov 03 12:24:50.096599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.096591    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d79af3-646b-4732-bff3-ff13a607ac9b" containerName="payload-pre-processing"
Nov 03 12:24:50.219009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.218972    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"]
Nov 03 12:24:50.223778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.219322    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.282979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.282170    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b6db28-openshift-default-dockercfg-jwclm\""
Nov 03 12:24:50.308714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.308676    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.322178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.322146    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.335825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.322434    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscs6\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.338122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.338090    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.338317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.338298    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.338491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.338472    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.338671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.338653    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.338955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.338937    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.339090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.339072    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440145    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440189    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440232    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscs6\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440321    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440369    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440419    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440482    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440541    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440783 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440576    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.440875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.440841    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.441151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.441102    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.441279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.441179    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.441345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.441277    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.441345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.441307    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.442870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.442846    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.443062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.443045    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.493084 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.493057    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.493216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.493197    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscs6\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6\") pod \"e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") " pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:50.651836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:50.651731    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:51.080582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.080539    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"]
Nov 03 12:24:51.084767 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.084713    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:51.085187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.084815    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:51.085187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.084858    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:51.292188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.292161    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"]
Nov 03 12:24:51.292952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.292933    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52bbfaa-1be7-40e7-9caa-997651d7155c" containerName="istio-proxy"
Nov 03 12:24:51.296415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.296398    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.345285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.345202    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-607dc3-openshift-default-dockercfg-xnbzk\""
Nov 03 12:24:51.394399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.394352    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"]
Nov 03 12:24:51.454252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454204    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454373    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454423    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454503    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454531    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454642    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454684    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67ts\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454806    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.454897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.454885    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556221    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556272    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67ts\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556318    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556380    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556416    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556481    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556629    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556668    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556703    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.556834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.556778    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.557123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.557099    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.557251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.557201    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.557321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.557304    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.557452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.557426    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.559377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.559348    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.559813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.559790    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.609650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.609564    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.609818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.609689    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67ts\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts\") pod \"e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") " pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:51.609870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:51.609817    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:52.032654 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:24:52.032621    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2575e087_47b8_4c58_8702_686f6de7673e.slice/crio-5dc8cd49b11b6645f4dcbc6039e9ce9ef697bfd8314759a332e5115ba010cc13 WatchSource:0}: Error finding container 5dc8cd49b11b6645f4dcbc6039e9ce9ef697bfd8314759a332e5115ba010cc13: Status 404 returned error can't find the container with id 5dc8cd49b11b6645f4dcbc6039e9ce9ef697bfd8314759a332e5115ba010cc13
Nov 03 12:24:52.032654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.032632    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"]
Nov 03 12:24:52.036002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.035962    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:52.036100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.036056    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:52.036144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.036115    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:24:52.049643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.049615    4546 generic.go:356] "Generic (PLEG): container finished" podID="867be5c5-4007-4c7b-b4f2-b89440379906" containerID="d30d4dd0a9ded501937b541b9d522827ca789e8014ef8d4e1c77df86e8db513a" exitCode=0
Nov 03 12:24:52.049772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.049679    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" event={"ID":"867be5c5-4007-4c7b-b4f2-b89440379906","Type":"ContainerDied","Data":"d30d4dd0a9ded501937b541b9d522827ca789e8014ef8d4e1c77df86e8db513a"}
Nov 03 12:24:52.049826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.049742    4546 scope.go:122] "RemoveContainer" containerID="aa284368b47faaeced64cf6cc6d1b1346da47749e86d88fb80be801d3bd65f01"
Nov 03 12:24:52.051725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.051696    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" event={"ID":"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a","Type":"ContainerStarted","Data":"ec07f906fefaa8b85ba5f4d7d7da39f074e623fc09ed9777876c609f9f17d74e"}
Nov 03 12:24:52.051904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.051784    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" event={"ID":"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a","Type":"ContainerStarted","Data":"d4d701afcc35d1bf8efbd06f549b86e8aea6cffcf017f85c0f3a11910bea9bda"}
Nov 03 12:24:52.053552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.053531    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" event={"ID":"2575e087-47b8-4c58-8702-686f6de7673e","Type":"ContainerStarted","Data":"5dc8cd49b11b6645f4dcbc6039e9ce9ef697bfd8314759a332e5115ba010cc13"}
Nov 03 12:24:52.061104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.061084    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:24:52.161190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.161123    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podStartSLOduration=2.161104359 podStartE2EDuration="2.161104359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:24:52.15855398 +0000 UTC m=+2033.730137893" watchObservedRunningTime="2026-11-03 12:24:52.161104359 +0000 UTC m=+2033.732688272"
Nov 03 12:24:52.163890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.163861    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.163917    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.163986    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164076    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164108    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164163    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164215    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:52.164278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164242    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164291    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.164445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.164383    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data\") pod \"867be5c5-4007-4c7b-b4f2-b89440379906\" (UID: \"867be5c5-4007-4c7b-b4f2-b89440379906\") "
Nov 03 12:24:52.165158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.165112    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:52.165273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.165166    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:52.165371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.165351    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:52.165433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.165411    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:52.165724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.165687    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.167115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.167050    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:52.167560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.167536    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "kube-api-access-nsqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:52.167719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.167701    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:52.168460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.168430    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy" pod "867be5c5-4007-4c7b-b4f2-b89440379906" (UID: "867be5c5-4007-4c7b-b4f2-b89440379906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:52.267852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.267801    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.267852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.267853    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/867be5c5-4007-4c7b-b4f2-b89440379906-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.268134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.267902    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.268134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.267918    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.268134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.267949    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsqt8\" (UniqueName: \"kubernetes.io/projected/867be5c5-4007-4c7b-b4f2-b89440379906-kube-api-access-nsqt8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.268134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.267963    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.268134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.268006    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/867be5c5-4007-4c7b-b4f2-b89440379906-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.268134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.268022    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/867be5c5-4007-4c7b-b4f2-b89440379906-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:52.651908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.651879    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:52.658035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.658006    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:52.764632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.764595    4546 patch_prober.go:29] interesting pod/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:24:52.764893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:52.764662    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:24:53.061658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.061557    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" event={"ID":"2575e087-47b8-4c58-8702-686f6de7673e","Type":"ContainerStarted","Data":"5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"}
Nov 03 12:24:53.063107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.063083    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"
Nov 03 12:24:53.063245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.063122    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g" event={"ID":"867be5c5-4007-4c7b-b4f2-b89440379906","Type":"ContainerDied","Data":"ba07cefca6b58b90c4ca9f6334072f47b276e8ca47ff5fe6eba4851594cac35c"}
Nov 03 12:24:53.063245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.063169    4546 scope.go:122] "RemoveContainer" containerID="d30d4dd0a9ded501937b541b9d522827ca789e8014ef8d4e1c77df86e8db513a"
Nov 03 12:24:53.063381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.063313    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:53.064596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.064558    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:24:53.172659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.172602    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podStartSLOduration=2.172586145 podStartE2EDuration="2.172586145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:24:53.169663585 +0000 UTC m=+2034.741247528" watchObservedRunningTime="2026-11-03 12:24:53.172586145 +0000 UTC m=+2034.744170057"
Nov 03 12:24:53.446595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.446475    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"]
Nov 03 12:24:53.449641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.449613    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-059838-openshift-default-b865b6f75-2hb6g"]
Nov 03 12:24:53.610645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.610609    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:53.616570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:53.616543    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:54.073675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.073586    4546 generic.go:356] "Generic (PLEG): container finished" podID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerID="802b3dedab59b0e7deed1c840878c01d36c8c35de62be2ed12a79c9d53db2a37" exitCode=0
Nov 03 12:24:54.073675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.073659    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" event={"ID":"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f","Type":"ContainerDied","Data":"802b3dedab59b0e7deed1c840878c01d36c8c35de62be2ed12a79c9d53db2a37"}
Nov 03 12:24:54.073919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.073719    4546 scope.go:122] "RemoveContainer" containerID="f36b9438b77c59ac0c3097abec447996adc816795a2c72e57169c4cf656d460c"
Nov 03 12:24:54.074197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.074180    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:54.075268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.075251    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:24:54.468668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.468635    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:24:54.573000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.572967    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" path="/var/lib/kubelet/pods/867be5c5-4007-4c7b-b4f2-b89440379906/volumes"
Nov 03 12:24:54.600727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.600694    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.600968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.600764    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.600968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.600898    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.600968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.600957    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.601199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601089    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.601199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601159    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:54.601199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601174    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.601363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601234    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.601363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601283    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.601468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601434    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:54.601699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601671    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:54.601928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601900    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:54.602081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.601966    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket\") pod \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\" (UID: \"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f\") "
Nov 03 12:24:54.602637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.602612    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.602782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.602648    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.602782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.602663    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.602782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.602677    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.602782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.602607    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:54.603543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.603521    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:54.604106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.604084    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:54.604210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.604101    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "kube-api-access-gp7gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:54.604842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.604824    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy" pod "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" (UID: "fe14a45a-e6d6-4e34-915a-92fd4ba3e56f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:54.703777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.703679    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.703777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.703709    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7gf\" (UniqueName: \"kubernetes.io/projected/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-kube-api-access-gp7gf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.703777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.703726    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.703777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.703737    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:54.703777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:54.703764    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:55.081601 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:55.081554    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts" event={"ID":"fe14a45a-e6d6-4e34-915a-92fd4ba3e56f","Type":"ContainerDied","Data":"4384a80267ca32919df2f1bc75e011d634b017c4a21ef127dad4f6579fba9fb1"}
Nov 03 12:24:55.081804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:55.081626    4546 scope.go:122] "RemoveContainer" containerID="802b3dedab59b0e7deed1c840878c01d36c8c35de62be2ed12a79c9d53db2a37"
Nov 03 12:24:55.081934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:55.081917    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"
Nov 03 12:24:55.237937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:55.237901    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"]
Nov 03 12:24:55.283762 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:55.283725    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-583bcf-openshift-default-ccb7cd54c-7kqts"]
Nov 03 12:24:56.573497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:56.573468    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" path="/var/lib/kubelet/pods/fe14a45a-e6d6-4e34-915a-92fd4ba3e56f/volumes"
Nov 03 12:24:58.466230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466199    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"]
Nov 03 12:24:58.466739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466726    4546 cpu_manager.go:414] "containerMap: removing container" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy"
Nov 03 12:24:58.466802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466740    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy"
Nov 03 12:24:58.466802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466786    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy"
Nov 03 12:24:58.466802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466791    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy"
Nov 03 12:24:58.466917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466816    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy"
Nov 03 12:24:58.466917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466822    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy"
Nov 03 12:24:58.466987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466935    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy"
Nov 03 12:24:58.466987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466947    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy"
Nov 03 12:24:58.466987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.466956    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy"
Nov 03 12:24:58.471856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.471830    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.608393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.608364    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b6db28\""
Nov 03 12:24:58.612491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.610147    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b6db28-dockercfg-mqffb\""
Nov 03 12:24:58.628792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.628688    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"]
Nov 03 12:24:58.635593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.635566    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"]
Nov 03 12:24:58.636597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.636574    4546 cpu_manager.go:414] "containerMap: removing container" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy"
Nov 03 12:24:58.636731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.636717    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867be5c5-4007-4c7b-b4f2-b89440379906" containerName="istio-proxy"
Nov 03 12:24:58.637090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.637071    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe14a45a-e6d6-4e34-915a-92fd4ba3e56f" containerName="istio-proxy"
Nov 03 12:24:58.641220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.641202    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:58.693943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.693909    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"]
Nov 03 12:24:58.715025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.714987    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjsd\" (UniqueName: \"kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd\") pod \"payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.715294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.715124    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.715384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.715330    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:58.715477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.715455    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlc8m\" (UniqueName: \"kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m\") pod \"payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:58.816257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.816166    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.816257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.816254    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:58.816508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.816295    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc8m\" (UniqueName: \"kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m\") pod \"payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:58.816508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.816395    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjsd\" (UniqueName: \"kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd\") pod \"payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.863769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.863723    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b6db28\""
Nov 03 12:24:58.867056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.867029    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:58.867177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.867160    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.869183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.869164    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjsd\" (UniqueName: \"kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd\") pod \"payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:58.869287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:58.869223    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlc8m\" (UniqueName: \"kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m\") pod \"payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:59.008085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:59.008059    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b6db28-dockercfg-mqffb\""
Nov 03 12:24:59.011207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:59.011188    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:24:59.094767 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:59.087183    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:24:59.382214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:59.382190    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"]
Nov 03 12:24:59.384696 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:24:59.384667    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c69c772_84a2_43f4_b07b_fa0bf4271a35.slice/crio-2ece4f663581d90d35dd4d2871bfeefa8a25261b20e2edb818bee924fe6e0fbe WatchSource:0}: Error finding container 2ece4f663581d90d35dd4d2871bfeefa8a25261b20e2edb818bee924fe6e0fbe: Status 404 returned error can't find the container with id 2ece4f663581d90d35dd4d2871bfeefa8a25261b20e2edb818bee924fe6e0fbe
Nov 03 12:24:59.475034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:24:59.475008    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"]
Nov 03 12:24:59.475967 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:24:59.475942    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83b5256_4e51_44ea_b27e_df501bb9fdcc.slice/crio-1a6f04cc9f257f73d21981deaf01ad0fd8abe8841f56c1a27f6e41671e502383 WatchSource:0}: Error finding container 1a6f04cc9f257f73d21981deaf01ad0fd8abe8841f56c1a27f6e41671e502383: Status 404 returned error can't find the container with id 1a6f04cc9f257f73d21981deaf01ad0fd8abe8841f56c1a27f6e41671e502383
Nov 03 12:25:00.118284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.118246    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr" event={"ID":"8c69c772-84a2-43f4-b07b-fa0bf4271a35","Type":"ContainerStarted","Data":"ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c"}
Nov 03 12:25:00.118523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.118315    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr" event={"ID":"8c69c772-84a2-43f4-b07b-fa0bf4271a35","Type":"ContainerStarted","Data":"2ece4f663581d90d35dd4d2871bfeefa8a25261b20e2edb818bee924fe6e0fbe"}
Nov 03 12:25:00.118523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.118461    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:25:00.119814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.119784    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67" event={"ID":"d83b5256-4e51-44ea-b27e-df501bb9fdcc","Type":"ContainerStarted","Data":"8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916"}
Nov 03 12:25:00.119928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.119838    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67" event={"ID":"d83b5256-4e51-44ea-b27e-df501bb9fdcc","Type":"ContainerStarted","Data":"1a6f04cc9f257f73d21981deaf01ad0fd8abe8841f56c1a27f6e41671e502383"}
Nov 03 12:25:00.119928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.119859    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:25:00.226847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.226790    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr" podStartSLOduration=2.226775506 podStartE2EDuration="2.226775506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:00.224179609 +0000 UTC m=+2041.795763521" watchObservedRunningTime="2026-11-03 12:25:00.226775506 +0000 UTC m=+2041.798359412"
Nov 03 12:25:00.330079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:00.330023    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67" podStartSLOduration=2.330009912 podStartE2EDuration="2.330009912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:00.328735388 +0000 UTC m=+2041.900319311" watchObservedRunningTime="2026-11-03 12:25:00.330009912 +0000 UTC m=+2041.901593844"
Nov 03 12:25:02.929884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:02.929846    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"]
Nov 03 12:25:02.935931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:02.935906    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:02.966599 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:02.966562    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:02.966934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:02.966907    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvfhm\" (UniqueName: \"kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm\") pod \"payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:03.005775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:02.989766    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"]
Nov 03 12:25:03.041775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.038076    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-607dc3\""
Nov 03 12:25:03.041775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.038333    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-607dc3-dockercfg-qhxj7\""
Nov 03 12:25:03.093786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.082554    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvfhm\" (UniqueName: \"kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm\") pod \"payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:03.093786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.082662    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:03.093786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.083855    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:03.093786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.090786    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"]
Nov 03 12:25:03.108806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.101815    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.129807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.128122    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"]
Nov 03 12:25:03.137263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.137234    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvfhm\" (UniqueName: \"kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm\") pod \"payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:03.183631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.183546    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxndf\" (UniqueName: \"kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf\") pod \"payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.183820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.183667    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.250551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.250513    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:03.289277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.285899    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxndf\" (UniqueName: \"kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf\") pod \"payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.289277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.286057    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.289277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.287661    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.341335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.341308    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxndf\" (UniqueName: \"kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf\") pod \"payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.423713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.423684    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:03.624625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.624597    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"]
Nov 03 12:25:03.626069 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:25:03.626042    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5830ff9e_298e_4811_ada3_c1fa96932883.slice/crio-2846c22012768257757be1c2ef90b6ea708cfe02f142478fe4cdf6a345601235 WatchSource:0}: Error finding container 2846c22012768257757be1c2ef90b6ea708cfe02f142478fe4cdf6a345601235: Status 404 returned error can't find the container with id 2846c22012768257757be1c2ef90b6ea708cfe02f142478fe4cdf6a345601235
Nov 03 12:25:03.790743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:03.790715    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"]
Nov 03 12:25:03.792922 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:25:03.792858    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8033137b_1d3a_4a59_8cd4_1b8050f7914c.slice/crio-019b76f2a85bfaab47a8d8cec101437f2675a7bef130e83dae7466e1dfaab26d WatchSource:0}: Error finding container 019b76f2a85bfaab47a8d8cec101437f2675a7bef130e83dae7466e1dfaab26d: Status 404 returned error can't find the container with id 019b76f2a85bfaab47a8d8cec101437f2675a7bef130e83dae7466e1dfaab26d
Nov 03 12:25:04.146807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.146769    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w" event={"ID":"8033137b-1d3a-4a59-8cd4-1b8050f7914c","Type":"ContainerStarted","Data":"7993650b1d966c95213c2bf6aeee7e6070043ca00ca0d104debc8ff1038b7617"}
Nov 03 12:25:04.147332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.146826    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w" event={"ID":"8033137b-1d3a-4a59-8cd4-1b8050f7914c","Type":"ContainerStarted","Data":"019b76f2a85bfaab47a8d8cec101437f2675a7bef130e83dae7466e1dfaab26d"}
Nov 03 12:25:04.147332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.146990    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:04.148505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.148477    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9" event={"ID":"5830ff9e-298e-4811-ada3-c1fa96932883","Type":"ContainerStarted","Data":"552a4d277ce13cdc5580beb76790f83ada811cbe96a4618d572ac6e8b8ec4421"}
Nov 03 12:25:04.148655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.148526    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:04.148655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.148542    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9" event={"ID":"5830ff9e-298e-4811-ada3-c1fa96932883","Type":"ContainerStarted","Data":"2846c22012768257757be1c2ef90b6ea708cfe02f142478fe4cdf6a345601235"}
Nov 03 12:25:04.253687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.253637    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w" podStartSLOduration=1.253622558 podStartE2EDuration="1.253622558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:04.251182245 +0000 UTC m=+2045.822766158" watchObservedRunningTime="2026-11-03 12:25:04.253622558 +0000 UTC m=+2045.825206469"
Nov 03 12:25:04.358526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:04.358472    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9" podStartSLOduration=2.358455723 podStartE2EDuration="2.358455723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:04.35568546 +0000 UTC m=+2045.927269385" watchObservedRunningTime="2026-11-03 12:25:04.358455723 +0000 UTC m=+2045.930039674"
Nov 03 12:25:07.652067 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:25:07.652034    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:25:11.128445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:11.128416    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:25:11.128882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:11.128479    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:25:15.163924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:15.163895    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:25:15.164369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:15.163971    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:25:16.354300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.354267    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"]
Nov 03 12:25:16.358719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.358699    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.413803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.413330    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"]
Nov 03 12:25:16.461731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.456813    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgn4\" (UniqueName: \"kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4\") pod \"payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.461731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.456876    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.461731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.457273    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-980a31\""
Nov 03 12:25:16.461731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.457407    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-980a31-dockercfg-bdt2n\""
Nov 03 12:25:16.518863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.518818    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"]
Nov 03 12:25:16.522981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.522963    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.558081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.558045    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgn4\" (UniqueName: \"kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4\") pod \"payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.558242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.558086    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.558779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.558729    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.576555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.576514    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"]
Nov 03 12:25:16.611688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.611622    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgn4\" (UniqueName: \"kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4\") pod \"payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.659235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.659207    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9d8\" (UniqueName: \"kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8\") pod \"payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.659387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.659272    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.672956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.672933    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:16.760726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.760694    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.760960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.760944    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9d8\" (UniqueName: \"kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8\") pod \"payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.761616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.761592    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.814495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.814462    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9d8\" (UniqueName: \"kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8\") pod \"payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:16.836874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:16.836837    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:17.042180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.042151    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"]
Nov 03 12:25:17.045950 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:25:17.045909    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483e3a95_e8d9_4430_9716_965e72058b16.slice/crio-b8ca7fab6696b07b9b7f3a568916501927408e4367628cc63605eb95de30548f WatchSource:0}: Error finding container b8ca7fab6696b07b9b7f3a568916501927408e4367628cc63605eb95de30548f: Status 404 returned error can't find the container with id b8ca7fab6696b07b9b7f3a568916501927408e4367628cc63605eb95de30548f
Nov 03 12:25:17.204080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.204053    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"]
Nov 03 12:25:17.204565 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:25:17.204537    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1592b0c5_e2b0_4670_8145_50282a5ad537.slice/crio-fa6a59adbd0a6f6cac776567714a82799a0d1bb98a06962e062f2a5733aae4b0 WatchSource:0}: Error finding container fa6a59adbd0a6f6cac776567714a82799a0d1bb98a06962e062f2a5733aae4b0: Status 404 returned error can't find the container with id fa6a59adbd0a6f6cac776567714a82799a0d1bb98a06962e062f2a5733aae4b0
Nov 03 12:25:17.268270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.268198    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc" event={"ID":"1592b0c5-e2b0-4670-8145-50282a5ad537","Type":"ContainerStarted","Data":"fa6a59adbd0a6f6cac776567714a82799a0d1bb98a06962e062f2a5733aae4b0"}
Nov 03 12:25:17.270409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.270386    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9" event={"ID":"483e3a95-e8d9-4430-9716-965e72058b16","Type":"ContainerStarted","Data":"82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a"}
Nov 03 12:25:17.270506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.270431    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9" event={"ID":"483e3a95-e8d9-4430-9716-965e72058b16","Type":"ContainerStarted","Data":"b8ca7fab6696b07b9b7f3a568916501927408e4367628cc63605eb95de30548f"}
Nov 03 12:25:17.270894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.270870    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:17.380647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:17.380585    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9" podStartSLOduration=1.3805698849999999 podStartE2EDuration="1.380569885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:17.378461688 +0000 UTC m=+2058.950045614" watchObservedRunningTime="2026-11-03 12:25:17.380569885 +0000 UTC m=+2058.952153797"
Nov 03 12:25:18.283843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:18.278354    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc" event={"ID":"1592b0c5-e2b0-4670-8145-50282a5ad537","Type":"ContainerStarted","Data":"49fdfe30f250bf7e5310dd381cd09ea4023db14b3859e27ced70610916cd72b1"}
Nov 03 12:25:18.386199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:18.386150    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc" podStartSLOduration=2.3861357229999998 podStartE2EDuration="2.386135723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:18.383439998 +0000 UTC m=+2059.955023911" watchObservedRunningTime="2026-11-03 12:25:18.386135723 +0000 UTC m=+2059.957719635"
Nov 03 12:25:19.284737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:19.284707    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:28.284738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:28.284706    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:25:30.294108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:30.294079    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:25:37.651833 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:25:37.651798    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:25:39.458468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:39.458432    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"]
Nov 03 12:25:39.556653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:39.556619    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"]
Nov 03 12:25:44.459515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.459473    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9" podUID="ee614dc0-0626-4c29-9952-c0fc0c7ead47" containerName="payload-processing" containerID="cri-o://dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf" gracePeriod=25
Nov 03 12:25:44.559912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.559855    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft" podUID="9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" containerName="payload-pre-processing" containerID="cri-o://a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16" gracePeriod=25
Nov 03 12:25:44.732863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.732834    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:25:44.801280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.801252    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume\") pod \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") "
Nov 03 12:25:44.801501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.801483    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw\" (UniqueName: \"kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw\") pod \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\" (UID: \"ee614dc0-0626-4c29-9952-c0fc0c7ead47\") "
Nov 03 12:25:44.801686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.801653    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume" pod "ee614dc0-0626-4c29-9952-c0fc0c7ead47" (UID: "ee614dc0-0626-4c29-9952-c0fc0c7ead47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:25:44.802071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.802046    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee614dc0-0626-4c29-9952-c0fc0c7ead47-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.803825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.803801    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw" pod "ee614dc0-0626-4c29-9952-c0fc0c7ead47" (UID: "ee614dc0-0626-4c29-9952-c0fc0c7ead47"). InnerVolumeSpecName "kube-api-access-mdhvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:25:44.831565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.831543    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:25:44.903112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.903082    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume\") pod \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") "
Nov 03 12:25:44.903295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.903185    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph\" (UniqueName: \"kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph\") pod \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\" (UID: \"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb\") "
Nov 03 12:25:44.903695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.903525    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume" pod "9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" (UID: "9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:25:44.904214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.904109    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdhvw\" (UniqueName: \"kubernetes.io/projected/ee614dc0-0626-4c29-9952-c0fc0c7ead47-kube-api-access-mdhvw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.904214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.904142    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.905849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:44.905826    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph" pod "9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" (UID: "9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb"). InnerVolumeSpecName "kube-api-access-dlxph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:25:45.005561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.005526    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxph\" (UniqueName: \"kubernetes.io/projected/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb-kube-api-access-dlxph\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:45.437725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.437642    4546 generic.go:356] "Generic (PLEG): container finished" podID="9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" containerID="a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16" exitCode=0
Nov 03 12:25:45.437725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.437703    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"
Nov 03 12:25:45.437941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.437728    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft" event={"ID":"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb","Type":"ContainerDied","Data":"a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16"}
Nov 03 12:25:45.437941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.437825    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft" event={"ID":"9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb","Type":"ContainerDied","Data":"0a2ae498a71982e128418aada21566bfe3e3809caec3d3622d0bd204e5661e4b"}
Nov 03 12:25:45.437941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.437855    4546 scope.go:122] "RemoveContainer" containerID="a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16"
Nov 03 12:25:45.439222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.439198    4546 generic.go:356] "Generic (PLEG): container finished" podID="ee614dc0-0626-4c29-9952-c0fc0c7ead47" containerID="dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf" exitCode=0
Nov 03 12:25:45.439368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.439266    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"
Nov 03 12:25:45.439368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.439266    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9" event={"ID":"ee614dc0-0626-4c29-9952-c0fc0c7ead47","Type":"ContainerDied","Data":"dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf"}
Nov 03 12:25:45.439368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.439312    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9" event={"ID":"ee614dc0-0626-4c29-9952-c0fc0c7ead47","Type":"ContainerDied","Data":"77df98855940a98a24078fc182556c2875bbc4770bdcac8637d01eb654f3aea4"}
Nov 03 12:25:45.448872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.448848    4546 scope.go:122] "RemoveContainer" containerID="a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16"
Nov 03 12:25:45.449219 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:25:45.449196    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16\": container with ID starting with a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16 not found: ID does not exist" containerID="a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16"
Nov 03 12:25:45.449311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.449226    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16"} err="failed to get container status \"a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16\": rpc error: code = NotFound desc = could not find container \"a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16\": container with ID starting with a3be0b11baa21b36217c646a0086e053307f43e448226127d89ee5984d648e16 not found: ID does not exist"
Nov 03 12:25:45.449311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.449242    4546 scope.go:122] "RemoveContainer" containerID="dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf"
Nov 03 12:25:45.458605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.458585    4546 scope.go:122] "RemoveContainer" containerID="dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf"
Nov 03 12:25:45.458888 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:25:45.458869    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf\": container with ID starting with dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf not found: ID does not exist" containerID="dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf"
Nov 03 12:25:45.458966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.458898    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf"} err="failed to get container status \"dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf\": rpc error: code = NotFound desc = could not find container \"dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf\": container with ID starting with dac91c77cdcd86bc1c43296f40f9b40c21303dbab5c9e02994c2d1f215aebcaf not found: ID does not exist"
Nov 03 12:25:45.590764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.590718    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"]
Nov 03 12:25:45.636245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.636211    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-76cc4778d7-79rft"]
Nov 03 12:25:45.736048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.736022    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"]
Nov 03 12:25:45.741649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:45.741625    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-76c75c94ff-6rsz9"]
Nov 03 12:25:46.573131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:46.573104    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" path="/var/lib/kubelet/pods/9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb/volumes"
Nov 03 12:25:46.573526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:25:46.573513    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee614dc0-0626-4c29-9952-c0fc0c7ead47" path="/var/lib/kubelet/pods/ee614dc0-0626-4c29-9952-c0fc0c7ead47/volumes"
Nov 03 12:26:01.793655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.793619    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"]
Nov 03 12:26:01.794192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.794177    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ee614dc0-0626-4c29-9952-c0fc0c7ead47" containerName="payload-processing"
Nov 03 12:26:01.794192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.794192    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee614dc0-0626-4c29-9952-c0fc0c7ead47" containerName="payload-processing"
Nov 03 12:26:01.794277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.794207    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" containerName="payload-pre-processing"
Nov 03 12:26:01.794277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.794213    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" containerName="payload-pre-processing"
Nov 03 12:26:01.794359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.794335    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee614dc0-0626-4c29-9952-c0fc0c7ead47" containerName="payload-processing"
Nov 03 12:26:01.794359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.794342    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb1fdc7-db8f-4c09-8c29-f75e7cf301bb" containerName="payload-pre-processing"
Nov 03 12:26:01.799112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.799094    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881512    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxw8\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881585    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881630    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881676    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881705    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881826    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881872    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881957    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:01.908774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:01.881989    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.018451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.018415    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c02ae4-openshift-default-dockercfg-jmjj4\""
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.021609    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"]
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027164    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027217    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027337    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxw8\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027386    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027429    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027475    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027508    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027619    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.027664    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.028574    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.029208    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.036483    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.036741    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.042776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.037027    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.082325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.081906    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.101848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.096285    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxw8\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.101848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.099514    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.102650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.102598    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token\") pod \"e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") " pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.149055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.149017    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:02.658814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.644960    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"]
Nov 03 12:26:02.664497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.664465    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:02.664673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.664657    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:02.664816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:02.664802    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:03.583444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:03.583401    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" event={"ID":"b091315f-b67b-450d-8c6a-191c8e388320","Type":"ContainerStarted","Data":"bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"}
Nov 03 12:26:03.583963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:03.583463    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" event={"ID":"b091315f-b67b-450d-8c6a-191c8e388320","Type":"ContainerStarted","Data":"19fa16ef9a7b943c7f997d2a02b152ac67b9c386378d37e80658b6f5f9c21331"}
Nov 03 12:26:03.695801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:03.695739    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podStartSLOduration=2.695725988 podStartE2EDuration="2.695725988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:03.695149735 +0000 UTC m=+2105.266733659" watchObservedRunningTime="2026-11-03 12:26:03.695725988 +0000 UTC m=+2105.267309900"
Nov 03 12:26:04.155717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:04.155680    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:04.179398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:04.179369    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:04.589839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:04.589810    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:04.590720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:04.590702    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:26:07.655349 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:26:07.655257    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:26:08.182689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.181139    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": read tcp 10.133.0.2:47292->10.133.0.181:15021: read: connection reset by peer" start-of-body=
Nov 03 12:26:08.182689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.181206    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": read tcp 10.133.0.2:47292->10.133.0.181:15021: read: connection reset by peer"
Nov 03 12:26:08.627455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.627425    4546 generic.go:356] "Generic (PLEG): container finished" podID="2575e087-47b8-4c58-8702-686f6de7673e" containerID="5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6" exitCode=0
Nov 03 12:26:08.627708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.627507    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" event={"ID":"2575e087-47b8-4c58-8702-686f6de7673e","Type":"ContainerDied","Data":"5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"}
Nov 03 12:26:08.627994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.627974    4546 scope.go:122] "RemoveContainer" containerID="5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"
Nov 03 12:26:08.629638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.629610    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:08.629714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.629698    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:08.629714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.629706    4546 generic.go:356] "Generic (PLEG): container finished" podID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerID="ec07f906fefaa8b85ba5f4d7d7da39f074e623fc09ed9777876c609f9f17d74e" exitCode=0
Nov 03 12:26:08.629806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.629788    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:08.629844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.629801    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" event={"ID":"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a","Type":"ContainerDied","Data":"ec07f906fefaa8b85ba5f4d7d7da39f074e623fc09ed9777876c609f9f17d74e"}
Nov 03 12:26:08.630192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.630175    4546 scope.go:122] "RemoveContainer" containerID="ec07f906fefaa8b85ba5f4d7d7da39f074e623fc09ed9777876c609f9f17d74e"
Nov 03 12:26:08.631505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.631481    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:08.631576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.631554    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:08.631618 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:08.631593    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:09.075046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.075012    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:26:09.697799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.696261    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" event={"ID":"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a","Type":"ContainerStarted","Data":"b8d08c0cc2f30c03e0f019ca59ed8d38da6ef2d1509d786024b248bf3c28980f"}
Nov 03 12:26:09.705908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.705111    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" event={"ID":"2575e087-47b8-4c58-8702-686f6de7673e","Type":"ContainerStarted","Data":"2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c"}
Nov 03 12:26:09.968984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.968950    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"]
Nov 03 12:26:09.973254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.973234    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:09.976900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.976877    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:09.977012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:09.976932    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gxc\" (UniqueName: \"kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:10.027115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.027084    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c02ae4\""
Nov 03 12:26:10.030991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.030961    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c02ae4-dockercfg-qknr4\""
Nov 03 12:26:10.047725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.047688    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"]
Nov 03 12:26:10.092779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.079226    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:10.092779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.079309    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gxc\" (UniqueName: \"kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:10.105148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.105112    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:10.185034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.184998    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gxc\" (UniqueName: \"kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:10.266342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.266249    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"]
Nov 03 12:26:10.282805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.282614    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.283418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.283391    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"]
Nov 03 12:26:10.286999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.286897    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:10.465978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.465934    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m659s\" (UniqueName: \"kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s\") pod \"payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.466217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.466042    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.569298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.569199    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.569873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.569850    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.570103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.570085    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m659s\" (UniqueName: \"kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s\") pod \"payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.622776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.619080    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:26:10.646774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.631833    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 12:26:10.646774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.631887    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 12:26:10.646774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.639284    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m659s\" (UniqueName: \"kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s\") pod \"payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:10.654773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.652102    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:26:10.654773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.652216    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 12:26:10.654773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.652256    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 12:26:10.724775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.723643    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"]
Nov 03 12:26:10.899031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:10.899007    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:11.349036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.342498    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"]
Nov 03 12:26:11.627841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.627138    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 12:26:11.627841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.627198    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 12:26:11.659777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.656152    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 12:26:11.659777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.656212    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 12:26:11.774342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.774307    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk" event={"ID":"4f021be0-7c44-4d36-aeae-8903929441d9","Type":"ContainerStarted","Data":"d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d"}
Nov 03 12:26:11.774509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.774364    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk" event={"ID":"4f021be0-7c44-4d36-aeae-8903929441d9","Type":"ContainerStarted","Data":"ec30b361096e18cbce46782c1ff1b826a3a3e3376bbf01664eee137642e9144f"}
Nov 03 12:26:11.774509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.774393    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:11.776035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.776011    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7" event={"ID":"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3","Type":"ContainerStarted","Data":"b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2"}
Nov 03 12:26:11.776140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.776051    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7" event={"ID":"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3","Type":"ContainerStarted","Data":"ab0436683f11cdd2d084aa6ba557b6c24dbbccce6bf74aded92eec4b6b519cab"}
Nov 03 12:26:11.776140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.776105    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:11.889011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.888899    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk" podStartSLOduration=1.8888806219999998 podStartE2EDuration="1.888880622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:11.884363961 +0000 UTC m=+2113.455947874" watchObservedRunningTime="2026-11-03 12:26:11.888880622 +0000 UTC m=+2113.460464536"
Nov 03 12:26:11.992034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:11.991812    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7" podStartSLOduration=2.991797191 podStartE2EDuration="2.991797191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:11.990230608 +0000 UTC m=+2113.561814541" watchObservedRunningTime="2026-11-03 12:26:11.991797191 +0000 UTC m=+2113.563381106"
Nov 03 12:26:12.625774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:12.625145    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 12:26:12.625774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:12.625222    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 12:26:12.656775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:12.655067    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 12:26:12.656775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:12.655125    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 12:26:13.620780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:13.616431    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 12:26:13.620780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:13.616491    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 12:26:13.657232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:13.657116    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 12:26:13.657232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:13.657174    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 12:26:14.615115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:14.615078    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 12:26:14.615389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:14.615364    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 12:26:14.656842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:14.654838    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 12:26:14.656842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:14.654900    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 12:26:15.616307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:15.616278    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:26:15.616806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:15.616737    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:26:15.617455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:15.617438    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:26:15.656323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:15.656300    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:26:15.656586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:15.656570    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:26:15.657220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:15.657205    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:26:22.786838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:22.786810    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:26:22.787273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:22.787244    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:26:24.084531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.084501    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"]
Nov 03 12:26:24.088924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.088901    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.142912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.142883    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"]
Nov 03 12:26:24.185060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.185029    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d25504\""
Nov 03 12:26:24.185257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.185239    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d25504-dockercfg-6hvsh\""
Nov 03 12:26:24.191819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.191797    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b525x\" (UniqueName: \"kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x\") pod \"payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.191980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.191963    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.293659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.293630    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b525x\" (UniqueName: \"kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x\") pod \"payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.293836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.293744    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.294420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.294395    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.346859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.346798    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b525x\" (UniqueName: \"kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x\") pod \"payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.402322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.402300    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:24.765900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.765877    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"]
Nov 03 12:26:24.766856 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:26:24.766826    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33427bb_f815_41e9_8e3c_97b172353553.slice/crio-0d45056612d9a290550c917c5daadf63aed19639a37b001fec42432b44a4a1bc WatchSource:0}: Error finding container 0d45056612d9a290550c917c5daadf63aed19639a37b001fec42432b44a4a1bc: Status 404 returned error can't find the container with id 0d45056612d9a290550c917c5daadf63aed19639a37b001fec42432b44a4a1bc
Nov 03 12:26:24.864197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.864038    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq" event={"ID":"f33427bb-f815-41e9-8e3c-97b172353553","Type":"ContainerStarted","Data":"0d45056612d9a290550c917c5daadf63aed19639a37b001fec42432b44a4a1bc"}
Nov 03 12:26:24.920036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.920002    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"]
Nov 03 12:26:24.924193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.924173    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:24.977804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:24.977775    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"]
Nov 03 12:26:25.023621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.023540    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6sz6\" (UniqueName: \"kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6\") pod \"payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.023805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.023687    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.125414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.125387    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6sz6\" (UniqueName: \"kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6\") pod \"payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.125822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.125465    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.126177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.126156    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.178157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.178127    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6sz6\" (UniqueName: \"kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6\") pod \"payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.237653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.237627    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.604252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.604228    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"]
Nov 03 12:26:25.607192 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:26:25.607161    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e9dcc1_03d9_4784_a732_ef63aaf95c40.slice/crio-927cba1ed315c524714dadbe24db8d06e2753e2ccff96c6bba2ec6ab0f3de6be WatchSource:0}: Error finding container 927cba1ed315c524714dadbe24db8d06e2753e2ccff96c6bba2ec6ab0f3de6be: Status 404 returned error can't find the container with id 927cba1ed315c524714dadbe24db8d06e2753e2ccff96c6bba2ec6ab0f3de6be
Nov 03 12:26:25.871942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.871855    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv" event={"ID":"14e9dcc1-03d9-4784-a732-ef63aaf95c40","Type":"ContainerStarted","Data":"c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53"}
Nov 03 12:26:25.871942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.871910    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv" event={"ID":"14e9dcc1-03d9-4784-a732-ef63aaf95c40","Type":"ContainerStarted","Data":"927cba1ed315c524714dadbe24db8d06e2753e2ccff96c6bba2ec6ab0f3de6be"}
Nov 03 12:26:25.871942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.871936    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:25.873612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.873585    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq" event={"ID":"f33427bb-f815-41e9-8e3c-97b172353553","Type":"ContainerStarted","Data":"3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f"}
Nov 03 12:26:25.873734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.873672    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:25.979289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:25.979226    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv" podStartSLOduration=1.9792075329999999 podStartE2EDuration="1.979207533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:25.977671254 +0000 UTC m=+2127.549255207" watchObservedRunningTime="2026-11-03 12:26:25.979207533 +0000 UTC m=+2127.550791446"
Nov 03 12:26:26.085653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:26.085601    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq" podStartSLOduration=2.085586 podStartE2EDuration="2.085586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:26.083303285 +0000 UTC m=+2127.654887210" watchObservedRunningTime="2026-11-03 12:26:26.085586 +0000 UTC m=+2127.657169910"
Nov 03 12:26:36.882388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:36.882355    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:26:36.883029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:36.882465    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:26:37.651838 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:26:37.651803    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:26:45.663491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:45.663462    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-zmp2m_b10c10e5-e265-4bcf-b38a-1de049716594/payload-processing/0.log"
Nov 03 12:26:45.919493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:45.919413    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-zmp2m_b10c10e5-e265-4bcf-b38a-1de049716594/payload-processing/0.log"
Nov 03 12:26:46.160314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:46.160283    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv_14e9dcc1-03d9-4784-a732-ef63aaf95c40/payload-processing/0.log"
Nov 03 12:26:55.561686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.561651    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"]
Nov 03 12:26:55.569562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.569522    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.618719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.618690    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-04d73d-openshift-default-dockercfg-swmlj\""
Nov 03 12:26:55.668853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.668816    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"]
Nov 03 12:26:55.697988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.697953    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698099    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698229    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698286    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698321    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvl96\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698448    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698658    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698688    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.698830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.698740    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.808515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.808729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808558    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.808729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808587    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.808729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808661    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.808946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808822    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.808946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808906    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.809056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808956    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.809056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.808983    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.809056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.809010    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvl96\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.809660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.809631    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.810057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.810033    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.817181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.817065    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.818202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.818176    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.818857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.818790    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.831014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.825682    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.831014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.827550    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.866016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.865974    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.867191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.867164    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvl96\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96\") pod \"e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") " pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:55.884705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:55.884672    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:56.303508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:56.303449    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"]
Nov 03 12:26:56.304698 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:26:56.304642    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f37d690_8f54_4ec0_a146_0b8e7385c277.slice/crio-19e688b6a1418fe51325f1a8e6d1dd2234b8d82d2123e7621bbac319b8e6c224 WatchSource:0}: Error finding container 19e688b6a1418fe51325f1a8e6d1dd2234b8d82d2123e7621bbac319b8e6c224: Status 404 returned error can't find the container with id 19e688b6a1418fe51325f1a8e6d1dd2234b8d82d2123e7621bbac319b8e6c224
Nov 03 12:26:56.307372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:56.307341    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:56.307464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:56.307405    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:56.307464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:56.307434    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:26:57.067732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:57.067691    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" event={"ID":"8f37d690-8f54-4ec0-a146-0b8e7385c277","Type":"ContainerStarted","Data":"ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"}
Nov 03 12:26:57.068237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:57.067782    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" event={"ID":"8f37d690-8f54-4ec0-a146-0b8e7385c277","Type":"ContainerStarted","Data":"19e688b6a1418fe51325f1a8e6d1dd2234b8d82d2123e7621bbac319b8e6c224"}
Nov 03 12:26:57.209775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:57.208566    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podStartSLOduration=2.208547262 podStartE2EDuration="2.208547262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:57.194297148 +0000 UTC m=+2158.765881061" watchObservedRunningTime="2026-11-03 12:26:57.208547262 +0000 UTC m=+2158.780131174"
Nov 03 12:26:57.885628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:57.885591    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:57.891213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:57.891189    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:58.083971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:58.083934    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:26:58.085162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:26:58.085139    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:27:03.439536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.439503    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"]
Nov 03 12:27:03.447775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.445675    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.498232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.498203    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"]
Nov 03 12:27:03.508624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.508591    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9wd\" (UniqueName: \"kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd\") pod \"payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.508854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.508829    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.564778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.541700    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-04d73d-dockercfg-w4nrl\""
Nov 03 12:27:03.576470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.576440    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-04d73d\""
Nov 03 12:27:03.611398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.611360    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"]
Nov 03 12:27:03.611709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.611675    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.611923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.611901    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9wd\" (UniqueName: \"kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd\") pod \"payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.612850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.612823    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.616798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.616779    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.661239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.661206    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"]
Nov 03 12:27:03.665401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.665373    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9wd\" (UniqueName: \"kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd\") pod \"payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.713453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.713389    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpk56\" (UniqueName: \"kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56\") pod \"payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.713453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.713442    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.761184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.761160    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:03.815495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.815463    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpk56\" (UniqueName: \"kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56\") pod \"payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.815850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.815551    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.816569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.816543    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.868591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.868554    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpk56\" (UniqueName: \"kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56\") pod \"payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:03.930669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:03.930636    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:04.131017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:04.130992    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"]
Nov 03 12:27:04.132077 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:27:04.132049    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b02228d_367c_4f89_b550_c16ebe85c372.slice/crio-3da133f50272c752c50d4671e5853d60fc2d3f75f49af77c8b826327b03ba1f9 WatchSource:0}: Error finding container 3da133f50272c752c50d4671e5853d60fc2d3f75f49af77c8b826327b03ba1f9: Status 404 returned error can't find the container with id 3da133f50272c752c50d4671e5853d60fc2d3f75f49af77c8b826327b03ba1f9
Nov 03 12:27:04.295260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:04.295234    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"]
Nov 03 12:27:04.296308 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:27:04.296282    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9795ff78_c344_4d5c_8d95_ac2466e1b4ed.slice/crio-756f32cc53ecdfe764acbb323cacfc9bf8271bf256ff9dbc0888cb3b9ae639f6 WatchSource:0}: Error finding container 756f32cc53ecdfe764acbb323cacfc9bf8271bf256ff9dbc0888cb3b9ae639f6: Status 404 returned error can't find the container with id 756f32cc53ecdfe764acbb323cacfc9bf8271bf256ff9dbc0888cb3b9ae639f6
Nov 03 12:27:05.132130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.132090    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt" event={"ID":"1b02228d-367c-4f89-b550-c16ebe85c372","Type":"ContainerStarted","Data":"dc366965ab6ba8f638d7ac6ca45b17584d8cf968015c6d9d91a69fe2d5881307"}
Nov 03 12:27:05.132602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.132157    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt" event={"ID":"1b02228d-367c-4f89-b550-c16ebe85c372","Type":"ContainerStarted","Data":"3da133f50272c752c50d4671e5853d60fc2d3f75f49af77c8b826327b03ba1f9"}
Nov 03 12:27:05.132602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.132282    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:05.133800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.133775    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf" event={"ID":"9795ff78-c344-4d5c-8d95-ac2466e1b4ed","Type":"ContainerStarted","Data":"c66fc17e5d4009ef1533cf0f089901e9b7a744e8a4fa01f71afa5d142e71d2bd"}
Nov 03 12:27:05.133911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.133818    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf" event={"ID":"9795ff78-c344-4d5c-8d95-ac2466e1b4ed","Type":"ContainerStarted","Data":"756f32cc53ecdfe764acbb323cacfc9bf8271bf256ff9dbc0888cb3b9ae639f6"}
Nov 03 12:27:05.133911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.133854    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:05.240187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.240142    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt" podStartSLOduration=2.240128211 podStartE2EDuration="2.240128211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:27:05.237123637 +0000 UTC m=+2166.808707561" watchObservedRunningTime="2026-11-03 12:27:05.240128211 +0000 UTC m=+2166.811712122"
Nov 03 12:27:05.344988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:05.344931    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf" podStartSLOduration=2.344905536 podStartE2EDuration="2.344905536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:27:05.34189836 +0000 UTC m=+2166.913482291" watchObservedRunningTime="2026-11-03 12:27:05.344905536 +0000 UTC m=+2166.916489449"
Nov 03 12:27:07.652268 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:27:07.652232    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:27:16.142674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:16.142645    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:27:16.143098 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:16.142719    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:27:18.980154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:18.980123    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"]
Nov 03 12:27:23.980777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:23.980715    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f" podUID="86d8c249-c37f-42c4-a0f9-2e613b05e8e7" containerName="payload-pre-processing" containerID="cri-o://31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92" gracePeriod=25
Nov 03 12:27:24.263414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.263380    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:27:24.266259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.266232    4546 generic.go:356] "Generic (PLEG): container finished" podID="86d8c249-c37f-42c4-a0f9-2e613b05e8e7" containerID="31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92" exitCode=0
Nov 03 12:27:24.266467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.266336    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"
Nov 03 12:27:24.266675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.266334    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f" event={"ID":"86d8c249-c37f-42c4-a0f9-2e613b05e8e7","Type":"ContainerDied","Data":"31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92"}
Nov 03 12:27:24.266839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.266735    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f" event={"ID":"86d8c249-c37f-42c4-a0f9-2e613b05e8e7","Type":"ContainerDied","Data":"fdbdff9332193fd01addb0206d263f4682d47afe0391dedc9c3f4edfd8af9325"}
Nov 03 12:27:24.266839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.266789    4546 scope.go:122] "RemoveContainer" containerID="31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92"
Nov 03 12:27:24.280034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.279938    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n\" (UniqueName: \"kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n\") pod \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") "
Nov 03 12:27:24.280115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.280095    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume\") pod \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\" (UID: \"86d8c249-c37f-42c4-a0f9-2e613b05e8e7\") "
Nov 03 12:27:24.280176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.280120    4546 scope.go:122] "RemoveContainer" containerID="31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92"
Nov 03 12:27:24.280489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.280463    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume" pod "86d8c249-c37f-42c4-a0f9-2e613b05e8e7" (UID: "86d8c249-c37f-42c4-a0f9-2e613b05e8e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:27:24.280803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.280782    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:24.280890 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:27:24.280830    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92\": container with ID starting with 31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92 not found: ID does not exist" containerID="31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92"
Nov 03 12:27:24.280890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.280860    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92"} err="failed to get container status \"31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92\": rpc error: code = NotFound desc = could not find container \"31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92\": container with ID starting with 31adc992bdd8ade66e88186d98fef20ffe958eedaefe6839a0fa4b40dabddf92 not found: ID does not exist"
Nov 03 12:27:24.282189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.282172    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n" pod "86d8c249-c37f-42c4-a0f9-2e613b05e8e7" (UID: "86d8c249-c37f-42c4-a0f9-2e613b05e8e7"). InnerVolumeSpecName "kube-api-access-7767n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:24.381910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.381881    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7767n\" (UniqueName: \"kubernetes.io/projected/86d8c249-c37f-42c4-a0f9-2e613b05e8e7-kube-api-access-7767n\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:24.685622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.685584    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"]
Nov 03 12:27:24.689494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:24.689468    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c8ee66-69cc879dff-d2g7f"]
Nov 03 12:27:26.573950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:26.573922    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d8c249-c37f-42c4-a0f9-2e613b05e8e7" path="/var/lib/kubelet/pods/86d8c249-c37f-42c4-a0f9-2e613b05e8e7/volumes"
Nov 03 12:27:37.660824 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:27:37.660715    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:27:42.052321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:42.052278    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"]
Nov 03 12:27:42.098292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:42.098262    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8226ca55-lkknn"]
Nov 03 12:27:42.576926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:42.576890    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5fff96-cfca-4d24-8b0e-9b4091aa994f" path="/var/lib/kubelet/pods/8c5fff96-cfca-4d24-8b0e-9b4091aa994f/volumes"
Nov 03 12:27:45.422343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:45.422312    4546 generic.go:356] "Generic (PLEG): container finished" podID="b091315f-b67b-450d-8c6a-191c8e388320" containerID="bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889" exitCode=0
Nov 03 12:27:45.422805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:45.422382    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" event={"ID":"b091315f-b67b-450d-8c6a-191c8e388320","Type":"ContainerDied","Data":"bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"}
Nov 03 12:27:45.422805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:45.422712    4546 scope.go:122] "RemoveContainer" containerID="bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"
Nov 03 12:27:45.424161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:45.424132    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:27:45.424260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:45.424202    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:27:45.424260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:45.424234    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:27:46.438548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:46.438514    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" event={"ID":"b091315f-b67b-450d-8c6a-191c8e388320","Type":"ContainerStarted","Data":"7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b"}
Nov 03 12:27:47.150131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:47.150100    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:27:47.151130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:47.151101    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 12:27:47.151284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:47.151169    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 12:27:48.150371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:48.150330    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 12:27:48.150790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:48.150391    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 12:27:49.150362 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:49.150324    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 12:27:49.150550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:49.150381    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 12:27:49.590652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:49.590622    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:27:50.149857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:50.149824    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 12:27:50.150049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:50.149887    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 12:27:51.150567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:51.150526    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 12:27:51.151001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:51.150581    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 12:27:52.155028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:52.154996    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:27:52.156101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:52.156080    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:27:59.168090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:59.168060    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"]
Nov 03 12:27:59.168493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:27:59.168318    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" containerID="cri-o://452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2" gracePeriod=30
Nov 03 12:28:00.271836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:00.271802    4546 scope.go:122] "RemoveContainer" containerID="52cbf5d5a6f33709fe221ebce8e07eb5d8df9b175a49e0770c63e3d071c3d227"
Nov 03 12:28:01.066201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:01.066164    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:28:01.066377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:01.066227    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:28:04.440833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.440777    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:28:04.554118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554092    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554162    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554181    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554215    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554269    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554322    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554398    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554396    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:04.554511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554432    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554479    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs\") pod \"b90b2734-457b-454d-b91d-d10560505147\" (UID: \"b90b2734-457b-454d-b91d-d10560505147\") "
Nov 03 12:28:04.554808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554613    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:04.554951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554924    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:04.554951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554938    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:04.555070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.554957    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:04.555125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.555074    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.555125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.555093    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90b2734-457b-454d-b91d-d10560505147-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.555125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.555107    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.555125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.555121    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.555493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.555133    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.556832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.556806    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:04.557050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.557024    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "kube-api-access-rxnxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:04.557160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.557088    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:28:04.559646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.559624    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo" pod "b90b2734-457b-454d-b91d-d10560505147" (UID: "b90b2734-457b-454d-b91d-d10560505147"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:28:04.587639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.587616    4546 generic.go:356] "Generic (PLEG): container finished" podID="b90b2734-457b-454d-b91d-d10560505147" containerID="452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2" exitCode=0
Nov 03 12:28:04.587768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.587675    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"
Nov 03 12:28:04.587768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.587690    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" event={"ID":"b90b2734-457b-454d-b91d-d10560505147","Type":"ContainerDied","Data":"452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2"}
Nov 03 12:28:04.587768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.587737    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65" event={"ID":"b90b2734-457b-454d-b91d-d10560505147","Type":"ContainerDied","Data":"a1c0134100160c8362faf8e0d47781d044b75b7bdb556dd5bd11aa17059fde1a"}
Nov 03 12:28:04.587929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.587790    4546 scope.go:122] "RemoveContainer" containerID="452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2"
Nov 03 12:28:04.598522 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.598504    4546 scope.go:122] "RemoveContainer" containerID="16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"
Nov 03 12:28:04.609256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.609237    4546 scope.go:122] "RemoveContainer" containerID="452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2"
Nov 03 12:28:04.609603 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:04.609577    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2\": container with ID starting with 452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2 not found: ID does not exist" containerID="452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2"
Nov 03 12:28:04.609698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.609612    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2"} err="failed to get container status \"452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2\": rpc error: code = NotFound desc = could not find container \"452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2\": container with ID starting with 452446b7718e6f44ce6a50993fc86e5a7a866bbe44516549110d4708afb502c2 not found: ID does not exist"
Nov 03 12:28:04.609698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.609631    4546 scope.go:122] "RemoveContainer" containerID="16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"
Nov 03 12:28:04.609995 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:04.609971    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060\": container with ID starting with 16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060 not found: ID does not exist" containerID="16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"
Nov 03 12:28:04.610099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.610004    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060"} err="failed to get container status \"16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060\": rpc error: code = NotFound desc = could not find container \"16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060\": container with ID starting with 16fe226ced9edf84d760fd58d6dc97a6596456c0d2acd79b3d9d5587f8174060 not found: ID does not exist"
Nov 03 12:28:04.656382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.656359    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxnxc\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-kube-api-access-rxnxc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.656494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.656386    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90b2734-457b-454d-b91d-d10560505147-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.656494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.656396    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90b2734-457b-454d-b91d-d10560505147-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.656494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.656421    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90b2734-457b-454d-b91d-d10560505147-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:04.699414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.699355    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"]
Nov 03 12:28:04.703627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:04.703602    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c8ee66-openshift-default-7bcf96b57c-hrj65"]
Nov 03 12:28:05.455154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:05.455124    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"]
Nov 03 12:28:05.560638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:05.560609    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"]
Nov 03 12:28:06.574224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:06.574197    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90b2734-457b-454d-b91d-d10560505147" path="/var/lib/kubelet/pods/b90b2734-457b-454d-b91d-d10560505147/volumes"
Nov 03 12:28:07.651343 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:07.651310    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:28:10.456450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.456414    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc" podUID="1592b0c5-e2b0-4670-8145-50282a5ad537" containerName="payload-processing" containerID="cri-o://49fdfe30f250bf7e5310dd381cd09ea4023db14b3859e27ced70610916cd72b1" gracePeriod=25
Nov 03 12:28:10.562007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.561962    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9" podUID="483e3a95-e8d9-4430-9716-965e72058b16" containerName="payload-pre-processing" containerID="cri-o://82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a" gracePeriod=25
Nov 03 12:28:10.635332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.635301    4546 generic.go:356] "Generic (PLEG): container finished" podID="1592b0c5-e2b0-4670-8145-50282a5ad537" containerID="49fdfe30f250bf7e5310dd381cd09ea4023db14b3859e27ced70610916cd72b1" exitCode=0
Nov 03 12:28:10.635640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.635383    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc" event={"ID":"1592b0c5-e2b0-4670-8145-50282a5ad537","Type":"ContainerDied","Data":"49fdfe30f250bf7e5310dd381cd09ea4023db14b3859e27ced70610916cd72b1"}
Nov 03 12:28:10.754872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.754842    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:28:10.833719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.833690    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8\" (UniqueName: \"kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8\") pod \"1592b0c5-e2b0-4670-8145-50282a5ad537\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") "
Nov 03 12:28:10.833933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.833901    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume\") pod \"1592b0c5-e2b0-4670-8145-50282a5ad537\" (UID: \"1592b0c5-e2b0-4670-8145-50282a5ad537\") "
Nov 03 12:28:10.834243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.834213    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume" pod "1592b0c5-e2b0-4670-8145-50282a5ad537" (UID: "1592b0c5-e2b0-4670-8145-50282a5ad537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:10.834353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.834335    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1592b0c5-e2b0-4670-8145-50282a5ad537-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:10.836061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.836035    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8" pod "1592b0c5-e2b0-4670-8145-50282a5ad537" (UID: "1592b0c5-e2b0-4670-8145-50282a5ad537"). InnerVolumeSpecName "kube-api-access-5w9d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:10.855689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.855670    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:28:10.934905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.934871    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4\" (UniqueName: \"kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4\") pod \"483e3a95-e8d9-4430-9716-965e72058b16\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") "
Nov 03 12:28:10.935155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.935134    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume\") pod \"483e3a95-e8d9-4430-9716-965e72058b16\" (UID: \"483e3a95-e8d9-4430-9716-965e72058b16\") "
Nov 03 12:28:10.935519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.935488    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume" pod "483e3a95-e8d9-4430-9716-965e72058b16" (UID: "483e3a95-e8d9-4430-9716-965e72058b16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:10.935646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.935590    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9d8\" (UniqueName: \"kubernetes.io/projected/1592b0c5-e2b0-4670-8145-50282a5ad537-kube-api-access-5w9d8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:10.935646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.935605    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483e3a95-e8d9-4430-9716-965e72058b16-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:10.937180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:10.937162    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4" pod "483e3a95-e8d9-4430-9716-965e72058b16" (UID: "483e3a95-e8d9-4430-9716-965e72058b16"). InnerVolumeSpecName "kube-api-access-hrgn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:11.036594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.036509    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrgn4\" (UniqueName: \"kubernetes.io/projected/483e3a95-e8d9-4430-9716-965e72058b16-kube-api-access-hrgn4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:11.644646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.644604    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"
Nov 03 12:28:11.645172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.644638    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc" event={"ID":"1592b0c5-e2b0-4670-8145-50282a5ad537","Type":"ContainerDied","Data":"fa6a59adbd0a6f6cac776567714a82799a0d1bb98a06962e062f2a5733aae4b0"}
Nov 03 12:28:11.645172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.644702    4546 scope.go:122] "RemoveContainer" containerID="49fdfe30f250bf7e5310dd381cd09ea4023db14b3859e27ced70610916cd72b1"
Nov 03 12:28:11.646910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.646888    4546 generic.go:356] "Generic (PLEG): container finished" podID="483e3a95-e8d9-4430-9716-965e72058b16" containerID="82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a" exitCode=0
Nov 03 12:28:11.647024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.646935    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9" event={"ID":"483e3a95-e8d9-4430-9716-965e72058b16","Type":"ContainerDied","Data":"82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a"}
Nov 03 12:28:11.647024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.646950    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"
Nov 03 12:28:11.647024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.646981    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9" event={"ID":"483e3a95-e8d9-4430-9716-965e72058b16","Type":"ContainerDied","Data":"b8ca7fab6696b07b9b7f3a568916501927408e4367628cc63605eb95de30548f"}
Nov 03 12:28:11.655933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.655909    4546 scope.go:122] "RemoveContainer" containerID="82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a"
Nov 03 12:28:11.665121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.665104    4546 scope.go:122] "RemoveContainer" containerID="82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a"
Nov 03 12:28:11.665382 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:11.665360    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a\": container with ID starting with 82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a not found: ID does not exist" containerID="82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a"
Nov 03 12:28:11.665424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.665391    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a"} err="failed to get container status \"82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a\": rpc error: code = NotFound desc = could not find container \"82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a\": container with ID starting with 82afb2737c06ee974edca6a9de22ae1276955ee464381686009791d63e706d7a not found: ID does not exist"
Nov 03 12:28:11.812106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.812009    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"]
Nov 03 12:28:11.857798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.857764    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-980a31-67657d4b87-qzms9"]
Nov 03 12:28:11.927382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.927342    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"]
Nov 03 12:28:11.930651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:11.930623    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-980a31-6467995db5-42fhc"]
Nov 03 12:28:12.623038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:12.622989    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1592b0c5-e2b0-4670-8145-50282a5ad537" path="/var/lib/kubelet/pods/1592b0c5-e2b0-4670-8145-50282a5ad537/volumes"
Nov 03 12:28:12.623773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:12.623734    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483e3a95-e8d9-4430-9716-965e72058b16" path="/var/lib/kubelet/pods/483e3a95-e8d9-4430-9716-965e72058b16/volumes"
Nov 03 12:28:20.893492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:20.893448    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"]
Nov 03 12:28:20.991333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:20.991301    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"]
Nov 03 12:28:25.894564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:25.894502    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk" podUID="4f021be0-7c44-4d36-aeae-8903929441d9" containerName="payload-processing" containerID="cri-o://d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d" gracePeriod=25
Nov 03 12:28:25.992546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:25.992493    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7" podUID="fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" containerName="payload-pre-processing" containerID="cri-o://b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2" gracePeriod=25
Nov 03 12:28:26.174956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.174927    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:28:26.242235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.242211    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:28:26.246466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.246448    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume\") pod \"4f021be0-7c44-4d36-aeae-8903929441d9\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") "
Nov 03 12:28:26.246540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.246500    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s\" (UniqueName: \"kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s\") pod \"4f021be0-7c44-4d36-aeae-8903929441d9\" (UID: \"4f021be0-7c44-4d36-aeae-8903929441d9\") "
Nov 03 12:28:26.246812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.246792    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume" pod "4f021be0-7c44-4d36-aeae-8903929441d9" (UID: "4f021be0-7c44-4d36-aeae-8903929441d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:26.247159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.247143    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f021be0-7c44-4d36-aeae-8903929441d9-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:26.248690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.248670    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s" pod "4f021be0-7c44-4d36-aeae-8903929441d9" (UID: "4f021be0-7c44-4d36-aeae-8903929441d9"). InnerVolumeSpecName "kube-api-access-m659s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:26.348540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.348517    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc\" (UniqueName: \"kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc\") pod \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") "
Nov 03 12:28:26.348801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.348783    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume\") pod \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\" (UID: \"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3\") "
Nov 03 12:28:26.349129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.349105    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume" pod "fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" (UID: "fe26311d-3649-4080-b1e3-8ffbdeb9e6d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:26.349292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.349277    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m659s\" (UniqueName: \"kubernetes.io/projected/4f021be0-7c44-4d36-aeae-8903929441d9-kube-api-access-m659s\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:26.349344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.349299    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:26.350677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.350658    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc" pod "fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" (UID: "fe26311d-3649-4080-b1e3-8ffbdeb9e6d3"). InnerVolumeSpecName "kube-api-access-77gxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:26.450881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.450791    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77gxc\" (UniqueName: \"kubernetes.io/projected/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3-kube-api-access-77gxc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:26.756313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.756287    4546 generic.go:356] "Generic (PLEG): container finished" podID="fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" containerID="b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2" exitCode=0
Nov 03 12:28:26.756506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.756333    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"
Nov 03 12:28:26.756506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.756378    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7" event={"ID":"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3","Type":"ContainerDied","Data":"b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2"}
Nov 03 12:28:26.756506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.756435    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7" event={"ID":"fe26311d-3649-4080-b1e3-8ffbdeb9e6d3","Type":"ContainerDied","Data":"ab0436683f11cdd2d084aa6ba557b6c24dbbccce6bf74aded92eec4b6b519cab"}
Nov 03 12:28:26.756506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.756466    4546 scope.go:122] "RemoveContainer" containerID="b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2"
Nov 03 12:28:26.758250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.758227    4546 generic.go:356] "Generic (PLEG): container finished" podID="4f021be0-7c44-4d36-aeae-8903929441d9" containerID="d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d" exitCode=0
Nov 03 12:28:26.758384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.758283    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"
Nov 03 12:28:26.758384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.758291    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk" event={"ID":"4f021be0-7c44-4d36-aeae-8903929441d9","Type":"ContainerDied","Data":"d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d"}
Nov 03 12:28:26.758384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.758328    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk" event={"ID":"4f021be0-7c44-4d36-aeae-8903929441d9","Type":"ContainerDied","Data":"ec30b361096e18cbce46782c1ff1b826a3a3e3376bbf01664eee137642e9144f"}
Nov 03 12:28:26.766217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.766191    4546 scope.go:122] "RemoveContainer" containerID="b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2"
Nov 03 12:28:26.766461 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:26.766441    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2\": container with ID starting with b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2 not found: ID does not exist" containerID="b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2"
Nov 03 12:28:26.766538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.766472    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2"} err="failed to get container status \"b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2\": rpc error: code = NotFound desc = could not find container \"b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2\": container with ID starting with b15f549201f2c9c917560cb6e6998ce6fda4ee8ae9a4c9290ca690739f8eb1e2 not found: ID does not exist"
Nov 03 12:28:26.766538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.766494    4546 scope.go:122] "RemoveContainer" containerID="d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d"
Nov 03 12:28:26.774807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.774790    4546 scope.go:122] "RemoveContainer" containerID="d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d"
Nov 03 12:28:26.775049 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:26.775028    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d\": container with ID starting with d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d not found: ID does not exist" containerID="d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d"
Nov 03 12:28:26.775112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.775059    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d"} err="failed to get container status \"d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d\": rpc error: code = NotFound desc = could not find container \"d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d\": container with ID starting with d4bb08a5b1c6b8a0dba0cd4c42d9b36ed9a5ae23bcac2babd5eeef04cc486d7d not found: ID does not exist"
Nov 03 12:28:26.909468 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.909440    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"]
Nov 03 12:28:26.954730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:26.954706    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-756fdf7f8c-bw7w7"]
Nov 03 12:28:27.055228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:27.055170    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"]
Nov 03 12:28:27.061056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:27.061035    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-6678b7b99-lhxbk"]
Nov 03 12:28:28.575191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:28.575163    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f021be0-7c44-4d36-aeae-8903929441d9" path="/var/lib/kubelet/pods/4f021be0-7c44-4d36-aeae-8903929441d9/volumes"
Nov 03 12:28:28.575556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:28.575535    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" path="/var/lib/kubelet/pods/fe26311d-3649-4080-b1e3-8ffbdeb9e6d3/volumes"
Nov 03 12:28:33.922514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.922482    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"]
Nov 03 12:28:33.923166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923151    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy"
Nov 03 12:28:33.923218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923167    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy"
Nov 03 12:28:33.923218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923185    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4f021be0-7c44-4d36-aeae-8903929441d9" containerName="payload-processing"
Nov 03 12:28:33.923218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923190    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f021be0-7c44-4d36-aeae-8903929441d9" containerName="payload-processing"
Nov 03 12:28:33.923218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923203    4546 cpu_manager.go:414] "containerMap: removing container" podUID="483e3a95-e8d9-4430-9716-965e72058b16" containerName="payload-pre-processing"
Nov 03 12:28:33.923218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923208    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483e3a95-e8d9-4430-9716-965e72058b16" containerName="payload-pre-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923233    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1592b0c5-e2b0-4670-8145-50282a5ad537" containerName="payload-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923237    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1592b0c5-e2b0-4670-8145-50282a5ad537" containerName="payload-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923246    4546 cpu_manager.go:414] "containerMap: removing container" podUID="86d8c249-c37f-42c4-a0f9-2e613b05e8e7" containerName="payload-pre-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923250    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d8c249-c37f-42c4-a0f9-2e613b05e8e7" containerName="payload-pre-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923263    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" containerName="payload-pre-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923269    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" containerName="payload-pre-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923347    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923355    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1592b0c5-e2b0-4670-8145-50282a5ad537" containerName="payload-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923361    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d8c249-c37f-42c4-a0f9-2e613b05e8e7" containerName="payload-pre-processing"
Nov 03 12:28:33.923371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923370    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe26311d-3649-4080-b1e3-8ffbdeb9e6d3" containerName="payload-pre-processing"
Nov 03 12:28:33.923708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923377    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy"
Nov 03 12:28:33.923708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923384    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f021be0-7c44-4d36-aeae-8903929441d9" containerName="payload-processing"
Nov 03 12:28:33.923708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.923389    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="483e3a95-e8d9-4430-9716-965e72058b16" containerName="payload-pre-processing"
Nov 03 12:28:33.928311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.928292    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:33.980149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:33.980115    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"]
Nov 03 12:28:34.022858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.022832    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:28:34.022858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.022841    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:28:34.023093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.022892    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:28:34.023093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.022988    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:28:34.038305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.038278    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8s29\" (UniqueName: \"kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.038414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.038309    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.038414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.038406    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.139186 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.139154    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8s29\" (UniqueName: \"kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.139315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.139190    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.139315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.139276    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.140001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.139944    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.141677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.141656    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.191784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.191702    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8s29\" (UniqueName: \"kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29\") pod \"maas-api-revoke-sub-7f7a8e96-mzzw8\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.240693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.240670    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:34.610207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.610174    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"]
Nov 03 12:28:34.611369 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:28:34.611340    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5874736c_b1d2_4fc8_9763_4a6bd8f5bdad.slice/crio-f0a5ae5be17cacdd7d4bc9370fce36ff361f2d79b9ed21d510d461c14d448c3d WatchSource:0}: Error finding container f0a5ae5be17cacdd7d4bc9370fce36ff361f2d79b9ed21d510d461c14d448c3d: Status 404 returned error can't find the container with id f0a5ae5be17cacdd7d4bc9370fce36ff361f2d79b9ed21d510d461c14d448c3d
Nov 03 12:28:34.809587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:34.809554    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8" event={"ID":"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad","Type":"ContainerStarted","Data":"f0a5ae5be17cacdd7d4bc9370fce36ff361f2d79b9ed21d510d461c14d448c3d"}
Nov 03 12:28:35.817582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:35.817508    4546 generic.go:356] "Generic (PLEG): container finished" podID="5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" containerID="fa9d912967100db5fa96f4059d6b78c970bdd1f74904f42d82cd6f8aa5422d2c" exitCode=0
Nov 03 12:28:35.817980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:35.817586    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8" event={"ID":"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad","Type":"ContainerDied","Data":"fa9d912967100db5fa96f4059d6b78c970bdd1f74904f42d82cd6f8aa5422d2c"}
Nov 03 12:28:36.980361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:36.980339    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:37.074426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.074343    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca\") pod \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") "
Nov 03 12:28:37.074580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.074437    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca\") pod \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") "
Nov 03 12:28:37.074668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.074648    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29\" (UniqueName: \"kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29\") pod \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\" (UID: \"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad\") "
Nov 03 12:28:37.074820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.074799    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca" pod "5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" (UID: "5874736c-b1d2-4fc8-9763-4a6bd8f5bdad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:37.075119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.075098    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:37.076692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.076668    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca" pod "5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" (UID: "5874736c-b1d2-4fc8-9763-4a6bd8f5bdad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:28:37.076828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.076765    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29" pod "5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" (UID: "5874736c-b1d2-4fc8-9763-4a6bd8f5bdad"). InnerVolumeSpecName "kube-api-access-w8s29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:37.176196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.176166    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8s29\" (UniqueName: \"kubernetes.io/projected/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-kube-api-access-w8s29\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:37.176196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.176193    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:37.651850 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:28:37.651817    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:28:37.834374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.834349    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"
Nov 03 12:28:37.834575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.834386    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8" event={"ID":"5874736c-b1d2-4fc8-9763-4a6bd8f5bdad","Type":"ContainerDied","Data":"f0a5ae5be17cacdd7d4bc9370fce36ff361f2d79b9ed21d510d461c14d448c3d"}
Nov 03 12:28:37.834575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:37.834438    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a5ae5be17cacdd7d4bc9370fce36ff361f2d79b9ed21d510d461c14d448c3d"
Nov 03 12:28:52.279212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.279171    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"]
Nov 03 12:28:52.280089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.280066    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy"
Nov 03 12:28:52.280089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.280090    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90b2734-457b-454d-b91d-d10560505147" containerName="istio-proxy"
Nov 03 12:28:52.280270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.280113    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" containerName="revoke-keys"
Nov 03 12:28:52.280270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.280121    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" containerName="revoke-keys"
Nov 03 12:28:52.280370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.280285    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" containerName="revoke-keys"
Nov 03 12:28:52.283821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.283799    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.379061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.379020    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:28:52.379061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.379055    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:28:52.379324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.379141    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:28:52.379324 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.379200    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:28:52.382860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.382838    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"]
Nov 03 12:28:52.388737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.388718    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.388884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.388868    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.388934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.388913    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmw8\" (UniqueName: \"kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.490202 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.490161    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.490419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.490269    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.490419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.490313    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmw8\" (UniqueName: \"kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.490918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.490891    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.492852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.492828    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.543497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.543426    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmw8\" (UniqueName: \"kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8\") pod \"maas-api-revoke-sub-cf0c29bd-fnvcj\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.598395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.598367    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:52.969767 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:52.969726    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"]
Nov 03 12:28:52.971685 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:28:52.971655    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9883170b_24e0_4039_811e_c855731c4aae.slice/crio-8d756b8a18969330583d19b5e646c252b97807d54e9660e74fea264c572737af WatchSource:0}: Error finding container 8d756b8a18969330583d19b5e646c252b97807d54e9660e74fea264c572737af: Status 404 returned error can't find the container with id 8d756b8a18969330583d19b5e646c252b97807d54e9660e74fea264c572737af
Nov 03 12:28:53.972506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:53.972476    4546 generic.go:356] "Generic (PLEG): container finished" podID="9883170b-24e0-4039-811e-c855731c4aae" containerID="7e5024cb2c0e957dbc417782c4f81d2ac15e3b80ae2b4baba939d0453e6cec0c" exitCode=0
Nov 03 12:28:53.972891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:53.972564    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj" event={"ID":"9883170b-24e0-4039-811e-c855731c4aae","Type":"ContainerDied","Data":"7e5024cb2c0e957dbc417782c4f81d2ac15e3b80ae2b4baba939d0453e6cec0c"}
Nov 03 12:28:53.972891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:53.972614    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj" event={"ID":"9883170b-24e0-4039-811e-c855731c4aae","Type":"ContainerStarted","Data":"8d756b8a18969330583d19b5e646c252b97807d54e9660e74fea264c572737af"}
Nov 03 12:28:54.980435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:54.980403    4546 generic.go:356] "Generic (PLEG): container finished" podID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerID="ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df" exitCode=0
Nov 03 12:28:54.980939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:54.980486    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" event={"ID":"8f37d690-8f54-4ec0-a146-0b8e7385c277","Type":"ContainerDied","Data":"ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"}
Nov 03 12:28:54.981365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:54.981136    4546 scope.go:122] "RemoveContainer" containerID="ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"
Nov 03 12:28:54.982846 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:54.982816    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:28:54.982964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:54.982882    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:28:54.982964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:54.982915    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:28:55.147731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.147705    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:55.226816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.226783    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca\") pod \"9883170b-24e0-4039-811e-c855731c4aae\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") "
Nov 03 12:28:55.226997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.226966    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca\") pod \"9883170b-24e0-4039-811e-c855731c4aae\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") "
Nov 03 12:28:55.227067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.227018    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8\" (UniqueName: \"kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8\") pod \"9883170b-24e0-4039-811e-c855731c4aae\" (UID: \"9883170b-24e0-4039-811e-c855731c4aae\") "
Nov 03 12:28:55.227442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.227389    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca" pod "9883170b-24e0-4039-811e-c855731c4aae" (UID: "9883170b-24e0-4039-811e-c855731c4aae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:55.227589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.227548    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883170b-24e0-4039-811e-c855731c4aae-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:55.229394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.229367    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca" pod "9883170b-24e0-4039-811e-c855731c4aae" (UID: "9883170b-24e0-4039-811e-c855731c4aae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:28:55.229519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.229477    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8" pod "9883170b-24e0-4039-811e-c855731c4aae" (UID: "9883170b-24e0-4039-811e-c855731c4aae"). InnerVolumeSpecName "kube-api-access-bcmw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:55.328858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.328818    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcmw8\" (UniqueName: \"kubernetes.io/projected/9883170b-24e0-4039-811e-c855731c4aae-kube-api-access-bcmw8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:55.328858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.328847    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883170b-24e0-4039-811e-c855731c4aae-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:55.988549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.988509    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" event={"ID":"8f37d690-8f54-4ec0-a146-0b8e7385c277","Type":"ContainerStarted","Data":"2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec"}
Nov 03 12:28:55.989962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.989942    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"
Nov 03 12:28:55.990100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.989982    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj" event={"ID":"9883170b-24e0-4039-811e-c855731c4aae","Type":"ContainerDied","Data":"8d756b8a18969330583d19b5e646c252b97807d54e9660e74fea264c572737af"}
Nov 03 12:28:55.990100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:55.990027    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d756b8a18969330583d19b5e646c252b97807d54e9660e74fea264c572737af"
Nov 03 12:28:56.885730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:56.885699    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:28:56.886671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:56.886647    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 12:28:56.886726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:56.886711    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 12:28:57.886050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:57.886009    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 12:28:57.886535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:57.886069    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 12:28:58.083943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:58.083905    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:28:58.885328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:58.885286    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 12:28:58.885503 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:58.885347    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 12:28:59.886072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:59.886032    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 12:28:59.886547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:28:59.886097    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 12:29:00.305140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:00.305105    4546 scope.go:122] "RemoveContainer" containerID="a1042319bfe4f3b54461b8a007064532af90b8caca8d710a686bf4f72cfceff9"
Nov 03 12:29:00.885335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:00.885300    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 12:29:00.885560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:00.885360    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 12:29:01.890190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:01.890160    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:29:01.891260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:01.891242    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:29:07.651230 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:29:07.651149    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:29:08.589228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.589197    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"]
Nov 03 12:29:08.589853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.589839    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9883170b-24e0-4039-811e-c855731c4aae" containerName="revoke-keys"
Nov 03 12:29:08.589908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.589855    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9883170b-24e0-4039-811e-c855731c4aae" containerName="revoke-keys"
Nov 03 12:29:08.589995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.589983    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9883170b-24e0-4039-811e-c855731c4aae" containerName="revoke-keys"
Nov 03 12:29:08.595099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.595082    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.647314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.647279    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"]
Nov 03 12:29:08.689003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.688966    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:29:08.689545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.689006    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:29:08.689545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.689011    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:29:08.689830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.689800    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.689959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.689935    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrkq\" (UniqueName: \"kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.690172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.690150    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.690172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.690164    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:29:08.791558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.791522    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.791774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.791578    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrkq\" (UniqueName: \"kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.791774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.791655    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.792231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.792201    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.794238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.794217    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.844373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.844294    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrkq\" (UniqueName: \"kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq\") pod \"maas-api-revoke-sub-6ffa5710-w26g7\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:08.908823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:08.908791    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:09.275356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:09.275328    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"]
Nov 03 12:29:09.276283 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:09.276250    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded798a56_0953_400e_9364_fba55a30d4c4.slice/crio-308b43c56268c659d7d5bc6c99016aba2dc71357bb83303be3f18c82ae466b91 WatchSource:0}: Error finding container 308b43c56268c659d7d5bc6c99016aba2dc71357bb83303be3f18c82ae466b91: Status 404 returned error can't find the container with id 308b43c56268c659d7d5bc6c99016aba2dc71357bb83303be3f18c82ae466b91
Nov 03 12:29:10.117508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:10.117469    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7" event={"ID":"ed798a56-0953-400e-9364-fba55a30d4c4","Type":"ContainerStarted","Data":"8c6e614658fcc8c07c03ad08dc9a7b0061eb6502423b955ef305dc226e4b2be1"}
Nov 03 12:29:10.117946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:10.117537    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7" event={"ID":"ed798a56-0953-400e-9364-fba55a30d4c4","Type":"ContainerStarted","Data":"308b43c56268c659d7d5bc6c99016aba2dc71357bb83303be3f18c82ae466b91"}
Nov 03 12:29:10.248392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:10.240289    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7" podStartSLOduration=2.2402669570000002 podStartE2EDuration="2.240266957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:10.231190835 +0000 UTC m=+2291.802774768" watchObservedRunningTime="2026-11-03 12:29:10.240266957 +0000 UTC m=+2291.811850869"
Nov 03 12:29:11.129539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:11.129496    4546 generic.go:356] "Generic (PLEG): container finished" podID="ed798a56-0953-400e-9364-fba55a30d4c4" containerID="8c6e614658fcc8c07c03ad08dc9a7b0061eb6502423b955ef305dc226e4b2be1" exitCode=0
Nov 03 12:29:11.130137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:11.129597    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7" event={"ID":"ed798a56-0953-400e-9364-fba55a30d4c4","Type":"ContainerDied","Data":"8c6e614658fcc8c07c03ad08dc9a7b0061eb6502423b955ef305dc226e4b2be1"}
Nov 03 12:29:12.346559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.346532    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:12.449313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.449216    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq\" (UniqueName: \"kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq\") pod \"ed798a56-0953-400e-9364-fba55a30d4c4\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") "
Nov 03 12:29:12.449472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.449336    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca\") pod \"ed798a56-0953-400e-9364-fba55a30d4c4\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") "
Nov 03 12:29:12.449472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.449388    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca\") pod \"ed798a56-0953-400e-9364-fba55a30d4c4\" (UID: \"ed798a56-0953-400e-9364-fba55a30d4c4\") "
Nov 03 12:29:12.449808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.449743    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca" pod "ed798a56-0953-400e-9364-fba55a30d4c4" (UID: "ed798a56-0953-400e-9364-fba55a30d4c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:12.450053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.449969    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:12.451598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.451573    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca" pod "ed798a56-0953-400e-9364-fba55a30d4c4" (UID: "ed798a56-0953-400e-9364-fba55a30d4c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:12.451598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.451586    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq" pod "ed798a56-0953-400e-9364-fba55a30d4c4" (UID: "ed798a56-0953-400e-9364-fba55a30d4c4"). InnerVolumeSpecName "kube-api-access-pnrkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:12.551506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.551469    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnrkq\" (UniqueName: \"kubernetes.io/projected/ed798a56-0953-400e-9364-fba55a30d4c4-kube-api-access-pnrkq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:12.551667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:12.551512    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed798a56-0953-400e-9364-fba55a30d4c4-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:13.180791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:13.180764    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"
Nov 03 12:29:13.180791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:13.180780    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7" event={"ID":"ed798a56-0953-400e-9364-fba55a30d4c4","Type":"ContainerDied","Data":"308b43c56268c659d7d5bc6c99016aba2dc71357bb83303be3f18c82ae466b91"}
Nov 03 12:29:13.181086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:13.180829    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308b43c56268c659d7d5bc6c99016aba2dc71357bb83303be3f18c82ae466b91"
Nov 03 12:29:14.334214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:14.334182    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"]
Nov 03 12:29:14.433455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:14.433422    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"]
Nov 03 12:29:19.334818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.334743    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr" podUID="8c69c772-84a2-43f4-b07b-fa0bf4271a35" containerName="payload-processing" containerID="cri-o://ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c" gracePeriod=25
Nov 03 12:29:19.434636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.434579    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67" podUID="d83b5256-4e51-44ea-b27e-df501bb9fdcc" containerName="payload-pre-processing" containerID="cri-o://8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916" gracePeriod=25
Nov 03 12:29:19.623920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.622862    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:29:19.662316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.662283    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume\") pod \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") "
Nov 03 12:29:19.662533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.662392    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m\" (UniqueName: \"kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m\") pod \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\" (UID: \"8c69c772-84a2-43f4-b07b-fa0bf4271a35\") "
Nov 03 12:29:19.662674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.662651    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume" pod "8c69c772-84a2-43f4-b07b-fa0bf4271a35" (UID: "8c69c772-84a2-43f4-b07b-fa0bf4271a35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:19.663605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.663241    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c69c772-84a2-43f4-b07b-fa0bf4271a35-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:19.665282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.665252    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m" pod "8c69c772-84a2-43f4-b07b-fa0bf4271a35" (UID: "8c69c772-84a2-43f4-b07b-fa0bf4271a35"). InnerVolumeSpecName "kube-api-access-dlc8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:19.702067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.702045    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:29:19.764408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.764377    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd\" (UniqueName: \"kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd\") pod \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") "
Nov 03 12:29:19.764604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.764530    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume\") pod \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\" (UID: \"d83b5256-4e51-44ea-b27e-df501bb9fdcc\") "
Nov 03 12:29:19.764931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.764902    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume" pod "d83b5256-4e51-44ea-b27e-df501bb9fdcc" (UID: "d83b5256-4e51-44ea-b27e-df501bb9fdcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:19.765085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.765063    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlc8m\" (UniqueName: \"kubernetes.io/projected/8c69c772-84a2-43f4-b07b-fa0bf4271a35-kube-api-access-dlc8m\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:19.765133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.765092    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d83b5256-4e51-44ea-b27e-df501bb9fdcc-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:19.768711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.768686    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd" pod "d83b5256-4e51-44ea-b27e-df501bb9fdcc" (UID: "d83b5256-4e51-44ea-b27e-df501bb9fdcc"). InnerVolumeSpecName "kube-api-access-8kjsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:19.866107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:19.866025    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjsd\" (UniqueName: \"kubernetes.io/projected/d83b5256-4e51-44ea-b27e-df501bb9fdcc-kube-api-access-8kjsd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:20.228258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.228224    4546 generic.go:356] "Generic (PLEG): container finished" podID="8c69c772-84a2-43f4-b07b-fa0bf4271a35" containerID="ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c" exitCode=0
Nov 03 12:29:20.228437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.228279    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"
Nov 03 12:29:20.228437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.228277    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr" event={"ID":"8c69c772-84a2-43f4-b07b-fa0bf4271a35","Type":"ContainerDied","Data":"ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c"}
Nov 03 12:29:20.228437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.228351    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr" event={"ID":"8c69c772-84a2-43f4-b07b-fa0bf4271a35","Type":"ContainerDied","Data":"2ece4f663581d90d35dd4d2871bfeefa8a25261b20e2edb818bee924fe6e0fbe"}
Nov 03 12:29:20.228437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.228383    4546 scope.go:122] "RemoveContainer" containerID="ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c"
Nov 03 12:29:20.229913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.229888    4546 generic.go:356] "Generic (PLEG): container finished" podID="d83b5256-4e51-44ea-b27e-df501bb9fdcc" containerID="8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916" exitCode=0
Nov 03 12:29:20.230029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.229931    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"
Nov 03 12:29:20.230029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.229955    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67" event={"ID":"d83b5256-4e51-44ea-b27e-df501bb9fdcc","Type":"ContainerDied","Data":"8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916"}
Nov 03 12:29:20.230029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.229998    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67" event={"ID":"d83b5256-4e51-44ea-b27e-df501bb9fdcc","Type":"ContainerDied","Data":"1a6f04cc9f257f73d21981deaf01ad0fd8abe8841f56c1a27f6e41671e502383"}
Nov 03 12:29:20.238693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.238675    4546 scope.go:122] "RemoveContainer" containerID="ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c"
Nov 03 12:29:20.239119 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:29:20.239094    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c\": container with ID starting with ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c not found: ID does not exist" containerID="ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c"
Nov 03 12:29:20.239230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.239130    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c"} err="failed to get container status \"ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c\": rpc error: code = NotFound desc = could not find container \"ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c\": container with ID starting with ca158c2b95704e145807bcdc6ccf175ba2c8a881bed34d5245884f196e5d4d0c not found: ID does not exist"
Nov 03 12:29:20.239230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.239153    4546 scope.go:122] "RemoveContainer" containerID="8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916"
Nov 03 12:29:20.248499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.248480    4546 scope.go:122] "RemoveContainer" containerID="8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916"
Nov 03 12:29:20.251099 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:29:20.251074    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916\": container with ID starting with 8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916 not found: ID does not exist" containerID="8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916"
Nov 03 12:29:20.251196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.251108    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916"} err="failed to get container status \"8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916\": rpc error: code = NotFound desc = could not find container \"8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916\": container with ID starting with 8be5b6e38f9ae5079794d26baee8066ead0ffa34c38eb7d4b4183d70f3ac3916 not found: ID does not exist"
Nov 03 12:29:20.380850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.380817    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"]
Nov 03 12:29:20.384737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.384707    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b6db28-66fffc7b4c-chbhr"]
Nov 03 12:29:20.485654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.485552    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"]
Nov 03 12:29:20.491115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.491090    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6db28-86589bb55f-6sh67"]
Nov 03 12:29:20.574376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.574348    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c69c772-84a2-43f4-b07b-fa0bf4271a35" path="/var/lib/kubelet/pods/8c69c772-84a2-43f4-b07b-fa0bf4271a35/volumes"
Nov 03 12:29:20.574727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:20.574714    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83b5256-4e51-44ea-b27e-df501bb9fdcc" path="/var/lib/kubelet/pods/d83b5256-4e51-44ea-b27e-df501bb9fdcc/volumes"
Nov 03 12:29:22.338740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.338697    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"]
Nov 03 12:29:22.339541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339518    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8c69c772-84a2-43f4-b07b-fa0bf4271a35" containerName="payload-processing"
Nov 03 12:29:22.339642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339543    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c69c772-84a2-43f4-b07b-fa0bf4271a35" containerName="payload-processing"
Nov 03 12:29:22.339642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339568    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ed798a56-0953-400e-9364-fba55a30d4c4" containerName="revoke-keys"
Nov 03 12:29:22.339642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339575    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed798a56-0953-400e-9364-fba55a30d4c4" containerName="revoke-keys"
Nov 03 12:29:22.339642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339629    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d83b5256-4e51-44ea-b27e-df501bb9fdcc" containerName="payload-pre-processing"
Nov 03 12:29:22.339642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339638    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83b5256-4e51-44ea-b27e-df501bb9fdcc" containerName="payload-pre-processing"
Nov 03 12:29:22.339933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339850    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83b5256-4e51-44ea-b27e-df501bb9fdcc" containerName="payload-pre-processing"
Nov 03 12:29:22.339933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339869    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c69c772-84a2-43f4-b07b-fa0bf4271a35" containerName="payload-processing"
Nov 03 12:29:22.339933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.339881    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed798a56-0953-400e-9364-fba55a30d4c4" containerName="revoke-keys"
Nov 03 12:29:22.343318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.343297    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.392078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.392054    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c02ae4\""
Nov 03 12:29:22.410314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.410287    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.410476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.410446    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8tr\" (UniqueName: \"kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr\") pod \"payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.438735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.438713    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c02ae4-dockercfg-kfbcc\""
Nov 03 12:29:22.441825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.441805    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"]
Nov 03 12:29:22.511675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.511634    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8tr\" (UniqueName: \"kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr\") pod \"payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.511838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.511828    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.512457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.512439    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.564342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.564317    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8tr\" (UniqueName: \"kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr\") pod \"payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.657127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.657056    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:22.795922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.795890    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"]
Nov 03 12:29:22.801004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.800984    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:22.853701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.853676    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"]
Nov 03 12:29:22.916195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.916110    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:22.916195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:22.916172    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgkf\" (UniqueName: \"kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:23.018000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.017955    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:23.018180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.018013    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgkf\" (UniqueName: \"kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:23.018866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.018841    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:23.026126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.026099    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"]
Nov 03 12:29:23.027146 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:23.027118    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a78d15_f4aa_40af_a0a2_42fe843a6a6b.slice/crio-28ffe85c834d67530e0f24d1e4245f9a97a7c88db4604afcc1ec7192336035b5 WatchSource:0}: Error finding container 28ffe85c834d67530e0f24d1e4245f9a97a7c88db4604afcc1ec7192336035b5: Status 404 returned error can't find the container with id 28ffe85c834d67530e0f24d1e4245f9a97a7c88db4604afcc1ec7192336035b5
Nov 03 12:29:23.045277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.045252    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"]
Nov 03 12:29:23.048306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.048282    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a26d042c-cn7t7"]
Nov 03 12:29:23.071464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.071441    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgkf\" (UniqueName: \"kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf\") pod \"payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:23.115438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.115413    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:23.259257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.259223    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" event={"ID":"96a78d15-f4aa-40af-a0a2-42fe843a6a6b","Type":"ContainerStarted","Data":"16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889"}
Nov 03 12:29:23.259453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.259277    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" event={"ID":"96a78d15-f4aa-40af-a0a2-42fe843a6a6b","Type":"ContainerStarted","Data":"28ffe85c834d67530e0f24d1e4245f9a97a7c88db4604afcc1ec7192336035b5"}
Nov 03 12:29:23.259453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.259298    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:23.369798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.369724    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" podStartSLOduration=1.369707295 podStartE2EDuration="1.369707295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:23.365096433 +0000 UTC m=+2304.936680367" watchObservedRunningTime="2026-11-03 12:29:23.369707295 +0000 UTC m=+2304.941291210"
Nov 03 12:29:23.495774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:23.494641    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"]
Nov 03 12:29:23.505458 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:23.505421    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3633a9_8fc6_4d8d_9318_60089e2a56ac.slice/crio-63f41e4a67f3c8de6dbeb06cb1e2a2bcf73676452ff178c1cb55a5b7caccf75b WatchSource:0}: Error finding container 63f41e4a67f3c8de6dbeb06cb1e2a2bcf73676452ff178c1cb55a5b7caccf75b: Status 404 returned error can't find the container with id 63f41e4a67f3c8de6dbeb06cb1e2a2bcf73676452ff178c1cb55a5b7caccf75b
Nov 03 12:29:24.276656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.276616    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq" event={"ID":"6f3633a9-8fc6-4d8d-9318-60089e2a56ac","Type":"ContainerStarted","Data":"9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2"}
Nov 03 12:29:24.276898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.276714    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:24.276898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.276738    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq" event={"ID":"6f3633a9-8fc6-4d8d-9318-60089e2a56ac","Type":"ContainerStarted","Data":"63f41e4a67f3c8de6dbeb06cb1e2a2bcf73676452ff178c1cb55a5b7caccf75b"}
Nov 03 12:29:24.383678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.383623    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq" podStartSLOduration=2.383609306 podStartE2EDuration="2.383609306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:24.383256821 +0000 UTC m=+2305.954840732" watchObservedRunningTime="2026-11-03 12:29:24.383609306 +0000 UTC m=+2305.955193218"
Nov 03 12:29:24.573279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.573197    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcd6531-55b2-42b5-917b-43f5bc496c33" path="/var/lib/kubelet/pods/fdcd6531-55b2-42b5-917b-43f5bc496c33/volumes"
Nov 03 12:29:24.617338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.617309    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"]
Nov 03 12:29:24.621621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.621602    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.670309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.670283    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:29:24.670526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.670510    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:29:24.672650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.672627    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:29:24.672786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.672663    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:29:24.674258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.674233    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"]
Nov 03 12:29:24.739686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.739648    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.739889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.739809    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchzh\" (UniqueName: \"kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.739960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.739924    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.841196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.841121    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchzh\" (UniqueName: \"kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.841326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.841200    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.841326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.841315    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.841911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.841879    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.843686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.843665    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.894118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.894086    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchzh\" (UniqueName: \"kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh\") pod \"maas-api-revoke-sub-1e20bba5-sgxzl\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:24.936233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:24.936202    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:25.306094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:25.306067    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"]
Nov 03 12:29:25.307518 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:25.307491    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732af0de_28b2_46ab_8dd2_7193e46c584c.slice/crio-347537c9a37e6f069acb5d15abf78e791c222d97344e55f523b2f75b7d38ad20 WatchSource:0}: Error finding container 347537c9a37e6f069acb5d15abf78e791c222d97344e55f523b2f75b7d38ad20: Status 404 returned error can't find the container with id 347537c9a37e6f069acb5d15abf78e791c222d97344e55f523b2f75b7d38ad20
Nov 03 12:29:26.290178 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:26.290146    4546 generic.go:356] "Generic (PLEG): container finished" podID="732af0de-28b2-46ab-8dd2-7193e46c584c" containerID="adb1fda735947277a340b64a080c1bf90a0cacc7cfcfc8b1c24dee7417222a24" exitCode=0
Nov 03 12:29:26.290577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:26.290231    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl" event={"ID":"732af0de-28b2-46ab-8dd2-7193e46c584c","Type":"ContainerDied","Data":"adb1fda735947277a340b64a080c1bf90a0cacc7cfcfc8b1c24dee7417222a24"}
Nov 03 12:29:26.290577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:26.290277    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl" event={"ID":"732af0de-28b2-46ab-8dd2-7193e46c584c","Type":"ContainerStarted","Data":"347537c9a37e6f069acb5d15abf78e791c222d97344e55f523b2f75b7d38ad20"}
Nov 03 12:29:27.443984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.443956    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:27.575501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.575407    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca\") pod \"732af0de-28b2-46ab-8dd2-7193e46c584c\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") "
Nov 03 12:29:27.575684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.575625    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca\") pod \"732af0de-28b2-46ab-8dd2-7193e46c584c\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") "
Nov 03 12:29:27.575684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.575661    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh\" (UniqueName: \"kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh\") pod \"732af0de-28b2-46ab-8dd2-7193e46c584c\" (UID: \"732af0de-28b2-46ab-8dd2-7193e46c584c\") "
Nov 03 12:29:27.576077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.576048    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca" pod "732af0de-28b2-46ab-8dd2-7193e46c584c" (UID: "732af0de-28b2-46ab-8dd2-7193e46c584c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:27.576333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.576309    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:27.577842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.577816    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca" pod "732af0de-28b2-46ab-8dd2-7193e46c584c" (UID: "732af0de-28b2-46ab-8dd2-7193e46c584c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:27.578015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.577996    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh" pod "732af0de-28b2-46ab-8dd2-7193e46c584c" (UID: "732af0de-28b2-46ab-8dd2-7193e46c584c"). InnerVolumeSpecName "kube-api-access-jchzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:27.677381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.677339    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732af0de-28b2-46ab-8dd2-7193e46c584c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:27.677381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:27.677371    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jchzh\" (UniqueName: \"kubernetes.io/projected/732af0de-28b2-46ab-8dd2-7193e46c584c-kube-api-access-jchzh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:28.089796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:28.089744    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"]
Nov 03 12:29:28.186317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:28.186279    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"]
Nov 03 12:29:28.304679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:28.304651    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"
Nov 03 12:29:28.304920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:28.304651    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl" event={"ID":"732af0de-28b2-46ab-8dd2-7193e46c584c","Type":"ContainerDied","Data":"347537c9a37e6f069acb5d15abf78e791c222d97344e55f523b2f75b7d38ad20"}
Nov 03 12:29:28.304920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:28.304786    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347537c9a37e6f069acb5d15abf78e791c222d97344e55f523b2f75b7d38ad20"
Nov 03 12:29:33.090584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.090524    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w" podUID="8033137b-1d3a-4a59-8cd4-1b8050f7914c" containerName="payload-processing" containerID="cri-o://7993650b1d966c95213c2bf6aeee7e6070043ca00ca0d104debc8ff1038b7617" gracePeriod=25
Nov 03 12:29:33.187223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.187183    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9" podUID="5830ff9e-298e-4811-ada3-c1fa96932883" containerName="payload-pre-processing" containerID="cri-o://552a4d277ce13cdc5580beb76790f83ada811cbe96a4618d572ac6e8b8ec4421" gracePeriod=25
Nov 03 12:29:33.341259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.341057    4546 generic.go:356] "Generic (PLEG): container finished" podID="8033137b-1d3a-4a59-8cd4-1b8050f7914c" containerID="7993650b1d966c95213c2bf6aeee7e6070043ca00ca0d104debc8ff1038b7617" exitCode=0
Nov 03 12:29:33.341259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.341133    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w" event={"ID":"8033137b-1d3a-4a59-8cd4-1b8050f7914c","Type":"ContainerDied","Data":"7993650b1d966c95213c2bf6aeee7e6070043ca00ca0d104debc8ff1038b7617"}
Nov 03 12:29:33.342977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.342955    4546 generic.go:356] "Generic (PLEG): container finished" podID="5830ff9e-298e-4811-ada3-c1fa96932883" containerID="552a4d277ce13cdc5580beb76790f83ada811cbe96a4618d572ac6e8b8ec4421" exitCode=0
Nov 03 12:29:33.343099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.343067    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9" event={"ID":"5830ff9e-298e-4811-ada3-c1fa96932883","Type":"ContainerDied","Data":"552a4d277ce13cdc5580beb76790f83ada811cbe96a4618d572ac6e8b8ec4421"}
Nov 03 12:29:33.490463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.490432    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:29:33.511963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.511939    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:29:33.555867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.555829    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf\" (UniqueName: \"kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf\") pod \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") "
Nov 03 12:29:33.556054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.555980    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume\") pod \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\" (UID: \"8033137b-1d3a-4a59-8cd4-1b8050f7914c\") "
Nov 03 12:29:33.556348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.556322    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume" pod "8033137b-1d3a-4a59-8cd4-1b8050f7914c" (UID: "8033137b-1d3a-4a59-8cd4-1b8050f7914c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:33.558231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.558206    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf" pod "8033137b-1d3a-4a59-8cd4-1b8050f7914c" (UID: "8033137b-1d3a-4a59-8cd4-1b8050f7914c"). InnerVolumeSpecName "kube-api-access-vxndf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:33.657878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.657790    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume\") pod \"5830ff9e-298e-4811-ada3-c1fa96932883\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") "
Nov 03 12:29:33.657878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.657851    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm\" (UniqueName: \"kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm\") pod \"5830ff9e-298e-4811-ada3-c1fa96932883\" (UID: \"5830ff9e-298e-4811-ada3-c1fa96932883\") "
Nov 03 12:29:33.658174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.658131    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume" pod "5830ff9e-298e-4811-ada3-c1fa96932883" (UID: "5830ff9e-298e-4811-ada3-c1fa96932883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:33.658623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.658204    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxndf\" (UniqueName: \"kubernetes.io/projected/8033137b-1d3a-4a59-8cd4-1b8050f7914c-kube-api-access-vxndf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:33.658623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.658218    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8033137b-1d3a-4a59-8cd4-1b8050f7914c-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:33.660151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.660125    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm" pod "5830ff9e-298e-4811-ada3-c1fa96932883" (UID: "5830ff9e-298e-4811-ada3-c1fa96932883"). InnerVolumeSpecName "kube-api-access-cvfhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:33.759215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.759183    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5830ff9e-298e-4811-ada3-c1fa96932883-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:33.759215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:33.759215    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvfhm\" (UniqueName: \"kubernetes.io/projected/5830ff9e-298e-4811-ada3-c1fa96932883-kube-api-access-cvfhm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:34.279676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.279644    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:29:34.350885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.350846    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w" event={"ID":"8033137b-1d3a-4a59-8cd4-1b8050f7914c","Type":"ContainerDied","Data":"019b76f2a85bfaab47a8d8cec101437f2675a7bef130e83dae7466e1dfaab26d"}
Nov 03 12:29:34.350885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.350881    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"
Nov 03 12:29:34.351120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.350924    4546 scope.go:122] "RemoveContainer" containerID="7993650b1d966c95213c2bf6aeee7e6070043ca00ca0d104debc8ff1038b7617"
Nov 03 12:29:34.352370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.352352    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"
Nov 03 12:29:34.352466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.352385    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9" event={"ID":"5830ff9e-298e-4811-ada3-c1fa96932883","Type":"ContainerDied","Data":"2846c22012768257757be1c2ef90b6ea708cfe02f142478fe4cdf6a345601235"}
Nov 03 12:29:34.361461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.361438    4546 scope.go:122] "RemoveContainer" containerID="552a4d277ce13cdc5580beb76790f83ada811cbe96a4618d572ac6e8b8ec4421"
Nov 03 12:29:34.489808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.489775    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"]
Nov 03 12:29:34.494150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.494122    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-607dc3-5dcb8dc747-lvrm9"]
Nov 03 12:29:34.573331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.573297    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5830ff9e-298e-4811-ada3-c1fa96932883" path="/var/lib/kubelet/pods/5830ff9e-298e-4811-ada3-c1fa96932883/volumes"
Nov 03 12:29:34.597119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.597054    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"]
Nov 03 12:29:34.600613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:34.600585    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-607dc3-774b784469-4pf9w"]
Nov 03 12:29:35.285254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:35.285222    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:29:36.582813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:36.582767    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8033137b-1d3a-4a59-8cd4-1b8050f7914c" path="/var/lib/kubelet/pods/8033137b-1d3a-4a59-8cd4-1b8050f7914c/volumes"
Nov 03 12:29:37.656311 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:29:37.656276    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:29:38.757118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:38.757086    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6_f0592ed8-db55-4ca6-b7fa-7fd0608d59ef/payload-processing/0.log"
Nov 03 12:29:39.067447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:39.067368    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-zmp2m_b10c10e5-e265-4bcf-b38a-1de049716594/payload-processing/0.log"
Nov 03 12:29:39.315232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:39.315203    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-zmp2m_b10c10e5-e265-4bcf-b38a-1de049716594/payload-processing/0.log"
Nov 03 12:29:39.548846 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:39.548814    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv_14e9dcc1-03d9-4784-a732-ef63aaf95c40/payload-processing/0.log"
Nov 03 12:29:39.789651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:39.789617    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv_14e9dcc1-03d9-4784-a732-ef63aaf95c40/payload-processing/0.log"
Nov 03 12:29:40.573978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.573941    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"]
Nov 03 12:29:40.574554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574533    4546 cpu_manager.go:414] "containerMap: removing container" podUID="732af0de-28b2-46ab-8dd2-7193e46c584c" containerName="revoke-keys"
Nov 03 12:29:40.574659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574555    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732af0de-28b2-46ab-8dd2-7193e46c584c" containerName="revoke-keys"
Nov 03 12:29:40.574659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574625    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5830ff9e-298e-4811-ada3-c1fa96932883" containerName="payload-pre-processing"
Nov 03 12:29:40.574659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574635    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5830ff9e-298e-4811-ada3-c1fa96932883" containerName="payload-pre-processing"
Nov 03 12:29:40.574659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574647    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8033137b-1d3a-4a59-8cd4-1b8050f7914c" containerName="payload-processing"
Nov 03 12:29:40.574659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574655    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8033137b-1d3a-4a59-8cd4-1b8050f7914c" containerName="payload-processing"
Nov 03 12:29:40.574935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574822    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5830ff9e-298e-4811-ada3-c1fa96932883" containerName="payload-pre-processing"
Nov 03 12:29:40.574935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574835    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8033137b-1d3a-4a59-8cd4-1b8050f7914c" containerName="payload-processing"
Nov 03 12:29:40.574935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.574851    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="732af0de-28b2-46ab-8dd2-7193e46c584c" containerName="revoke-keys"
Nov 03 12:29:40.579658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.579637    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.629445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.629417    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"]
Nov 03 12:29:40.648204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.648174    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.648385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.648249    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.648448 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.648423    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8mj\" (UniqueName: \"kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.671977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.671942    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:29:40.671977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.671943    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:29:40.674207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.674182    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:29:40.674207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.674200    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:29:40.750267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.750235    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8mj\" (UniqueName: \"kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.750450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.750332    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.750450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.750426    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.750982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.750961    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.753194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.753167    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.802926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.802891    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8mj\" (UniqueName: \"kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj\") pod \"maas-api-revoke-sub-6d058b2c-kr6j6\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:40.894254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:40.894180    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:41.267772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:41.260244    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"]
Nov 03 12:29:41.404524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:41.404482    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6" event={"ID":"dd859100-f110-4739-84ae-ebedca482be9","Type":"ContainerStarted","Data":"4fa6c3d831a75e7980694ac1492c6653f1e1791460971187c608af85708cfbce"}
Nov 03 12:29:42.412701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:42.412667    4546 generic.go:356] "Generic (PLEG): container finished" podID="dd859100-f110-4739-84ae-ebedca482be9" containerID="aa07609eef0a7ad5b15c11ded1d04542ab9bcd768c6fcdf9b145bc28fb6c7b62" exitCode=0
Nov 03 12:29:42.413239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:42.412786    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6" event={"ID":"dd859100-f110-4739-84ae-ebedca482be9","Type":"ContainerDied","Data":"aa07609eef0a7ad5b15c11ded1d04542ab9bcd768c6fcdf9b145bc28fb6c7b62"}
Nov 03 12:29:43.579408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.579384    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:43.688737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.688568    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca\") pod \"dd859100-f110-4739-84ae-ebedca482be9\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") "
Nov 03 12:29:43.689192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.689162    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj\" (UniqueName: \"kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj\") pod \"dd859100-f110-4739-84ae-ebedca482be9\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") "
Nov 03 12:29:43.689327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.689310    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca\") pod \"dd859100-f110-4739-84ae-ebedca482be9\" (UID: \"dd859100-f110-4739-84ae-ebedca482be9\") "
Nov 03 12:29:43.689959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.689712    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca" pod "dd859100-f110-4739-84ae-ebedca482be9" (UID: "dd859100-f110-4739-84ae-ebedca482be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:43.690608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.690569    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd859100-f110-4739-84ae-ebedca482be9-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:43.691491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.691466    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca" pod "dd859100-f110-4739-84ae-ebedca482be9" (UID: "dd859100-f110-4739-84ae-ebedca482be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:43.691984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.691961    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj" pod "dd859100-f110-4739-84ae-ebedca482be9" (UID: "dd859100-f110-4739-84ae-ebedca482be9"). InnerVolumeSpecName "kube-api-access-xk8mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:43.791991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.791956    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd859100-f110-4739-84ae-ebedca482be9-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:43.791991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:43.791993    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk8mj\" (UniqueName: \"kubernetes.io/projected/dd859100-f110-4739-84ae-ebedca482be9-kube-api-access-xk8mj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:44.427104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:44.427074    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"
Nov 03 12:29:44.427295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:44.427107    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6" event={"ID":"dd859100-f110-4739-84ae-ebedca482be9","Type":"ContainerDied","Data":"4fa6c3d831a75e7980694ac1492c6653f1e1791460971187c608af85708cfbce"}
Nov 03 12:29:44.427295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:44.427157    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa6c3d831a75e7980694ac1492c6653f1e1791460971187c608af85708cfbce"
Nov 03 12:29:47.280944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.280903    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"]
Nov 03 12:29:47.281507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.281490    4546 cpu_manager.go:414] "containerMap: removing container" podUID="dd859100-f110-4739-84ae-ebedca482be9" containerName="revoke-keys"
Nov 03 12:29:47.281555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.281511    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd859100-f110-4739-84ae-ebedca482be9" containerName="revoke-keys"
Nov 03 12:29:47.281742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.281726    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd859100-f110-4739-84ae-ebedca482be9" containerName="revoke-keys"
Nov 03 12:29:47.285500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.285482    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.338659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.338626    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"]
Nov 03 12:29:47.348321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.348289    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.348649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.348623    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw9w\" (UniqueName: \"kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.381644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.381588    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2daac7-dockercfg-j59j8\""
Nov 03 12:29:47.382240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.382214    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2daac7\""
Nov 03 12:29:47.455647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.455604    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw9w\" (UniqueName: \"kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.457070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.457044    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.458433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.458075    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.478869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.478827    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"]
Nov 03 12:29:47.484012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.483986    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.509201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.509167    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw9w\" (UniqueName: \"kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.536235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.536159    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"]
Nov 03 12:29:47.559626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.559587    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.559971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.559736    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2fr\" (UniqueName: \"kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr\") pod \"payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.600872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.600840    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:47.662821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.661136    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.662821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.661285    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2fr\" (UniqueName: \"kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr\") pod \"payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.662821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.662376    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.715925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.715893    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2fr\" (UniqueName: \"kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr\") pod \"payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.798886 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.798800    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:47.976112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:47.976081    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"]
Nov 03 12:29:47.977741 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:47.977714    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e25fd51_8258_41b1_a7f5_19362f9af42f.slice/crio-6f82c6784350932ab0f27385ca3b08d208a601de98df1dde214e379de3a247a7 WatchSource:0}: Error finding container 6f82c6784350932ab0f27385ca3b08d208a601de98df1dde214e379de3a247a7: Status 404 returned error can't find the container with id 6f82c6784350932ab0f27385ca3b08d208a601de98df1dde214e379de3a247a7
Nov 03 12:29:48.166732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.166703    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"]
Nov 03 12:29:48.168200 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:48.168171    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13240456_69cd_444c_bffd_863a8e8f2bae.slice/crio-61a2c3869d1b82e9ec6af7f9d19e32f6605f02946308fb1afd11e0c345e6b1ee WatchSource:0}: Error finding container 61a2c3869d1b82e9ec6af7f9d19e32f6605f02946308fb1afd11e0c345e6b1ee: Status 404 returned error can't find the container with id 61a2c3869d1b82e9ec6af7f9d19e32f6605f02946308fb1afd11e0c345e6b1ee
Nov 03 12:29:48.466816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.466776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7" event={"ID":"13240456-69cd-444c-bffd-863a8e8f2bae","Type":"ContainerStarted","Data":"e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3"}
Nov 03 12:29:48.467282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.466846    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:48.467282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.466874    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7" event={"ID":"13240456-69cd-444c-bffd-863a8e8f2bae","Type":"ContainerStarted","Data":"61a2c3869d1b82e9ec6af7f9d19e32f6605f02946308fb1afd11e0c345e6b1ee"}
Nov 03 12:29:48.468454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.468433    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp" event={"ID":"0e25fd51-8258-41b1-a7f5-19362f9af42f","Type":"ContainerStarted","Data":"82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7"}
Nov 03 12:29:48.468545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.468478    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:48.468545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.468494    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp" event={"ID":"0e25fd51-8258-41b1-a7f5-19362f9af42f","Type":"ContainerStarted","Data":"6f82c6784350932ab0f27385ca3b08d208a601de98df1dde214e379de3a247a7"}
Nov 03 12:29:48.578143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.578080    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7" podStartSLOduration=1.578062179 podStartE2EDuration="1.578062179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:48.571908998 +0000 UTC m=+2330.143492910" watchObservedRunningTime="2026-11-03 12:29:48.578062179 +0000 UTC m=+2330.149646092"
Nov 03 12:29:48.742170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:48.742053    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp" podStartSLOduration=1.742038301 podStartE2EDuration="1.742038301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:48.74052691 +0000 UTC m=+2330.312110824" watchObservedRunningTime="2026-11-03 12:29:48.742038301 +0000 UTC m=+2330.313622210"
Nov 03 12:29:56.333595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.333556    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"]
Nov 03 12:29:56.339166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.339135    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.366144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.366115    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"]
Nov 03 12:29:56.370605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.370581    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.374956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.374922    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.374985    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375096    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375182    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375216    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375307    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk99x\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375341    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375476    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.375641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.375626    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.418069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.418034    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"]
Nov 03 12:29:56.419200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.419174    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:29:56.434199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.433980    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:29:56.434199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.434037    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:29:56.434199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.434089    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:29:56.471255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.471169    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"]
Nov 03 12:29:56.476617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476554    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476610    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476648    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476692    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476720    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476768    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk99x\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476796    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476843    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476901    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.476950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476945    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.477398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.476982    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tvg\" (UniqueName: \"kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.477398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.477082    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.477398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.477120    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.477563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.477480    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.477804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.477780    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.478036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.478015    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.478929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.478908    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.483797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.480387    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.483797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.480771    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.536909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.536859    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.539826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.539791    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk99x\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.578524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.578493    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.578724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.578547    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.578724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.578573    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tvg\" (UniqueName: \"kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.579177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.579156    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.581072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.581049    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.634140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.634061    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tvg\" (UniqueName: \"kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg\") pod \"maas-api-revoke-sub-00ddb7aa-tddx2\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:56.660790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.660728    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:56.687983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:56.687951    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:57.091403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.091374    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"]
Nov 03 12:29:57.092201 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:57.092158    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71da09f_6993_4c88_9e9f_ac5059b9a6fd.slice/crio-cfa7d12370f08c0de9158319521ef09942385b21712bf4286d4aae84499d48d2 WatchSource:0}: Error finding container cfa7d12370f08c0de9158319521ef09942385b21712bf4286d4aae84499d48d2: Status 404 returned error can't find the container with id cfa7d12370f08c0de9158319521ef09942385b21712bf4286d4aae84499d48d2
Nov 03 12:29:57.094881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.094850    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:29:57.094956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.094927    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:29:57.095002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.094958    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:29:57.134456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.134417    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"]
Nov 03 12:29:57.135296 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:29:57.135258    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f696ab_ccb6_4714_a8f3_42cfcc4593b4.slice/crio-104b37435ee246ea3a51a799c39b76ae71414e94b924e49c0751b09bd7dbc2e1 WatchSource:0}: Error finding container 104b37435ee246ea3a51a799c39b76ae71414e94b924e49c0751b09bd7dbc2e1: Status 404 returned error can't find the container with id 104b37435ee246ea3a51a799c39b76ae71414e94b924e49c0751b09bd7dbc2e1
Nov 03 12:29:57.545950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.545906    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" event={"ID":"a71da09f-6993-4c88-9e9f-ac5059b9a6fd","Type":"ContainerStarted","Data":"3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764"}
Nov 03 12:29:57.546465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.545999    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" event={"ID":"a71da09f-6993-4c88-9e9f-ac5059b9a6fd","Type":"ContainerStarted","Data":"cfa7d12370f08c0de9158319521ef09942385b21712bf4286d4aae84499d48d2"}
Nov 03 12:29:57.548126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.548090    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2" event={"ID":"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4","Type":"ContainerStarted","Data":"c7eb306f398d5abf760b0a4b11d1ce18326d946ea4d1ec821d98558227a20d8e"}
Nov 03 12:29:57.548267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.548144    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2" event={"ID":"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4","Type":"ContainerStarted","Data":"104b37435ee246ea3a51a799c39b76ae71414e94b924e49c0751b09bd7dbc2e1"}
Nov 03 12:29:57.661461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.661419    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:29:57.662646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.662604    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podStartSLOduration=1.662590381 podStartE2EDuration="1.662590381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:57.659291266 +0000 UTC m=+2339.230875180" watchObservedRunningTime="2026-11-03 12:29:57.662590381 +0000 UTC m=+2339.234174293"
Nov 03 12:29:57.769266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:57.769209    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2" podStartSLOduration=1.769194922 podStartE2EDuration="1.769194922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:57.766315568 +0000 UTC m=+2339.337899480" watchObservedRunningTime="2026-11-03 12:29:57.769194922 +0000 UTC m=+2339.340778834"
Nov 03 12:29:58.560355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:58.560319    4546 generic.go:356] "Generic (PLEG): container finished" podID="a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" containerID="c7eb306f398d5abf760b0a4b11d1ce18326d946ea4d1ec821d98558227a20d8e" exitCode=0
Nov 03 12:29:58.560877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:58.560404    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2" event={"ID":"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4","Type":"ContainerDied","Data":"c7eb306f398d5abf760b0a4b11d1ce18326d946ea4d1ec821d98558227a20d8e"}
Nov 03 12:29:58.662398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:58.662368    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 12:29:58.662561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:58.662416    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 12:29:59.477538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.477508    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:29:59.477714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.477578    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:29:59.661513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.661468    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 12:29:59.662065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.661570    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 12:29:59.841195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.841161    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:29:59.948704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.948672    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca\") pod \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") "
Nov 03 12:29:59.949110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.949066    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca\") pod \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") "
Nov 03 12:29:59.949252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.949129    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca" pod "a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" (UID: "a2f696ab-ccb6-4714-a8f3-42cfcc4593b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:59.949252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.949151    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg\" (UniqueName: \"kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg\") pod \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\" (UID: \"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4\") "
Nov 03 12:29:59.949971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.949916    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:59.951651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.951623    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca" pod "a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" (UID: "a2f696ab-ccb6-4714-a8f3-42cfcc4593b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:59.952026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:29:59.952006    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg" pod "a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" (UID: "a2f696ab-ccb6-4714-a8f3-42cfcc4593b4"). InnerVolumeSpecName "kube-api-access-b6tvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:00.051535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.051447    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:00.051535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.051482    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6tvg\" (UniqueName: \"kubernetes.io/projected/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4-kube-api-access-b6tvg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:00.158556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.158502    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"]
Nov 03 12:30:00.159431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.159412    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" containerName="revoke-keys"
Nov 03 12:30:00.159540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.159433    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" containerName="revoke-keys"
Nov 03 12:30:00.159618 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.159598    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" containerName="revoke-keys"
Nov 03 12:30:00.163589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.163567    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"]
Nov 03 12:30:00.163730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.163707    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.167941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.167924    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"]
Nov 03 12:30:00.168064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.168046    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.174768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.172716    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"]
Nov 03 12:30:00.174768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.172832    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.178764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.177780    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"]
Nov 03 12:30:00.178764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.177834    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.182705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.182579    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"]
Nov 03 12:30:00.182854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.182781    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.186913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.186894    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"]
Nov 03 12:30:00.187040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.187023    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.191268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.191252    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.239411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.239213    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"]
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.253549    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47mr\" (UniqueName: \"kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.253589    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.253857    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2r5x\" (UniqueName: \"kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x\") pod \"maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254549    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccr9q\" (UniqueName: \"kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254604    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254652    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254781    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254836    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6lfs\" (UniqueName: \"kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ldp\" (UniqueName: \"kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp\") pod \"maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.254987    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8h78\" (UniqueName: \"kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78\") pod \"maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.255075    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt766\" (UniqueName: \"kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.255125    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.255397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.255156    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.265799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.264974    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"]
Nov 03 12:30:00.274785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.270801    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"]
Nov 03 12:30:00.336109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.336023    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"]
Nov 03 12:30:00.343841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.341622    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"]
Nov 03 12:30:00.351252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.351229    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"]
Nov 03 12:30:00.355136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.355112    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"]
Nov 03 12:30:00.356252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.356227    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt766\" (UniqueName: \"kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.356447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.356427    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.356583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.356567    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.356869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.356851    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47mr\" (UniqueName: \"kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.357011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.356994    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.357149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357131    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2r5x\" (UniqueName: \"kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x\") pod \"maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.357294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357275    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccr9q\" (UniqueName: \"kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.357430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357413    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.357583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357555    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.357664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357562    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.357736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357665    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.357736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357699    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6lfs\" (UniqueName: \"kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.357903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357733    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.357903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357776    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.357903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357849    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ldp\" (UniqueName: \"kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp\") pod \"maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.357903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357869    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8h78\" (UniqueName: \"kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78\") pod \"maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.358451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.358423    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.358686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.357080    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.358850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.358828    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.359244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.359224    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.359348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.359229    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.435626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.435589    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ldp\" (UniqueName: \"kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp\") pod \"maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.435824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.435639    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47mr\" (UniqueName: \"kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.435990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.435958    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt766\" (UniqueName: \"kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.436285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.436266    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8h78\" (UniqueName: \"kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78\") pod \"maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.436595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.436578    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2r5x\" (UniqueName: \"kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x\") pod \"maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.436674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.436630    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccr9q\" (UniqueName: \"kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.438565 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.438543    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6lfs\" (UniqueName: \"kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.482816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.482786    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:00.490363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.490344    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:00.505862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.505833    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:00.531823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.522978    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:00.536976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.536956    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:00.554198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.554054    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:00.570736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.570677    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:00.664593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.664138    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 12:30:00.664593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.664191    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 12:30:00.682055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.682019    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2" event={"ID":"a2f696ab-ccb6-4714-a8f3-42cfcc4593b4","Type":"ContainerDied","Data":"104b37435ee246ea3a51a799c39b76ae71414e94b924e49c0751b09bd7dbc2e1"}
Nov 03 12:30:00.682179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.682086    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="104b37435ee246ea3a51a799c39b76ae71414e94b924e49c0751b09bd7dbc2e1"
Nov 03 12:30:00.682218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.682201    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"
Nov 03 12:30:00.900360 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:00.900329    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"]
Nov 03 12:30:00.901921 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:00.901891    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adc3273_1a5e_4ebc_94a0_c1dd4331d401.slice/crio-7c725926486182a6cd796057907d8dc4992b6f84b9a1f7304fb8884e446adebe WatchSource:0}: Error finding container 7c725926486182a6cd796057907d8dc4992b6f84b9a1f7304fb8884e446adebe: Status 404 returned error can't find the container with id 7c725926486182a6cd796057907d8dc4992b6f84b9a1f7304fb8884e446adebe
Nov 03 12:30:01.116782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.116178    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"]
Nov 03 12:30:01.177560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.177531    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"]
Nov 03 12:30:01.178136 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:01.178107    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33258f10_6d09_46aa_9b61_f928198a9e6e.slice/crio-2e6777912d244c4aacf1d411a9eadd448cd72dc513c48c5c7a1940923cebe981 WatchSource:0}: Error finding container 2e6777912d244c4aacf1d411a9eadd448cd72dc513c48c5c7a1940923cebe981: Status 404 returned error can't find the container with id 2e6777912d244c4aacf1d411a9eadd448cd72dc513c48c5c7a1940923cebe981
Nov 03 12:30:01.234189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.234161    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"]
Nov 03 12:30:01.258347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.258322    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"]
Nov 03 12:30:01.285613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.285591    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"]
Nov 03 12:30:01.337900 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:01.337812    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec63eb2_c4da_4174_9ed8_9bb86d19b60d.slice/crio-283b66402d27b0af1686d03df99bcebe485746ad2a115934823aa9f0becb964c WatchSource:0}: Error finding container 283b66402d27b0af1686d03df99bcebe485746ad2a115934823aa9f0becb964c: Status 404 returned error can't find the container with id 283b66402d27b0af1686d03df99bcebe485746ad2a115934823aa9f0becb964c
Nov 03 12:30:01.339335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.339305    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"]
Nov 03 12:30:01.436625 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:01.436586    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332bd850_8555_4835_84c5_1bc013fa34ff.slice/crio-2cfe98b135c4d7bcff461848dae712fab56cff7cc63c62127de5cdb804809949 WatchSource:0}: Error finding container 2cfe98b135c4d7bcff461848dae712fab56cff7cc63c62127de5cdb804809949: Status 404 returned error can't find the container with id 2cfe98b135c4d7bcff461848dae712fab56cff7cc63c62127de5cdb804809949
Nov 03 12:30:01.442817 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:01.442791    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd1cde6_9c5d_42b2_99b5_43daad17bcff.slice/crio-110e5c6bb13f85cb09abdc94b8babe180cf9f4333f360fa9044e2e0744b837fe WatchSource:0}: Error finding container 110e5c6bb13f85cb09abdc94b8babe180cf9f4333f360fa9044e2e0744b837fe: Status 404 returned error can't find the container with id 110e5c6bb13f85cb09abdc94b8babe180cf9f4333f360fa9044e2e0744b837fe
Nov 03 12:30:01.661727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.661702    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 12:30:01.661839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.661743    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 12:30:01.696945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.695942    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" event={"ID":"3b0a4c6d-3f96-4746-ad49-2fb1981a623e","Type":"ContainerStarted","Data":"054da918d82ea5bf79c4b8b9b8f4b6c6fe4392f31c23887cd40f3ce56b20aabc"}
Nov 03 12:30:01.697962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.697936    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" event={"ID":"332bd850-8555-4835-84c5-1bc013fa34ff","Type":"ContainerStarted","Data":"2cfe98b135c4d7bcff461848dae712fab56cff7cc63c62127de5cdb804809949"}
Nov 03 12:30:01.699457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.699430    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" event={"ID":"7287e5ec-18c1-4d81-b3d9-44939c635b39","Type":"ContainerStarted","Data":"692a6084d9ee74650c918d6396e31958fbd3f920f27bbcc371b3e61b7e2169a4"}
Nov 03 12:30:01.701238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.701217    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerStarted","Data":"a98400c2889616cff61794eeba5bcef09dad0d846f521b8cf9a0744de8010e83"}
Nov 03 12:30:01.701355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.701256    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerStarted","Data":"2e6777912d244c4aacf1d411a9eadd448cd72dc513c48c5c7a1940923cebe981"}
Nov 03 12:30:01.702727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.702702    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" event={"ID":"aec63eb2-c4da-4174-9ed8-9bb86d19b60d","Type":"ContainerStarted","Data":"283b66402d27b0af1686d03df99bcebe485746ad2a115934823aa9f0becb964c"}
Nov 03 12:30:01.704017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.703989    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" event={"ID":"cbd1cde6-9c5d-42b2-99b5-43daad17bcff","Type":"ContainerStarted","Data":"110e5c6bb13f85cb09abdc94b8babe180cf9f4333f360fa9044e2e0744b837fe"}
Nov 03 12:30:01.705486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.705439    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerStarted","Data":"64fe2efc9b0cf9f90ec24c3f31585fa8fa013a95a5be2e441dc3df803b3be81a"}
Nov 03 12:30:01.705571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.705503    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerStarted","Data":"7c725926486182a6cd796057907d8dc4992b6f84b9a1f7304fb8884e446adebe"}
Nov 03 12:30:01.840182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.839906    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" podStartSLOduration=1.839883306 podStartE2EDuration="1.839883306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:01.805704089 +0000 UTC m=+2343.377288001" watchObservedRunningTime="2026-11-03 12:30:01.839883306 +0000 UTC m=+2343.411467218"
Nov 03 12:30:01.938806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:01.938289    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" podStartSLOduration=1.938271867 podStartE2EDuration="1.938271867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:01.918234836 +0000 UTC m=+2343.489818769" watchObservedRunningTime="2026-11-03 12:30:01.938271867 +0000 UTC m=+2343.509855801"
Nov 03 12:30:02.661834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.661797    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 12:30:02.661983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.661854    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 12:30:02.738578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.738553    4546 generic.go:356] "Generic (PLEG): container finished" podID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerID="486abbaba814744b7ee0096428ab02842ad981508164db2d9bd5b516b3c768d1" exitCode=60
Nov 03 12:30:02.738967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.738631    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" event={"ID":"aec63eb2-c4da-4174-9ed8-9bb86d19b60d","Type":"ContainerDied","Data":"486abbaba814744b7ee0096428ab02842ad981508164db2d9bd5b516b3c768d1"}
Nov 03 12:30:02.738967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.738941    4546 scope.go:122] "RemoveContainer" containerID="486abbaba814744b7ee0096428ab02842ad981508164db2d9bd5b516b3c768d1"
Nov 03 12:30:02.740410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.740390    4546 generic.go:356] "Generic (PLEG): container finished" podID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerID="3093933905d3788fce7456adbf7068e02309d9ef6d310e748645876f0f23f319" exitCode=60
Nov 03 12:30:02.740538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.740466    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" event={"ID":"cbd1cde6-9c5d-42b2-99b5-43daad17bcff","Type":"ContainerDied","Data":"3093933905d3788fce7456adbf7068e02309d9ef6d310e748645876f0f23f319"}
Nov 03 12:30:02.740708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.740691    4546 scope.go:122] "RemoveContainer" containerID="3093933905d3788fce7456adbf7068e02309d9ef6d310e748645876f0f23f319"
Nov 03 12:30:02.742622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.742553    4546 generic.go:356] "Generic (PLEG): container finished" podID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerID="64fe2efc9b0cf9f90ec24c3f31585fa8fa013a95a5be2e441dc3df803b3be81a" exitCode=60
Nov 03 12:30:02.742622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.742601    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerDied","Data":"64fe2efc9b0cf9f90ec24c3f31585fa8fa013a95a5be2e441dc3df803b3be81a"}
Nov 03 12:30:02.742800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.742785    4546 scope.go:122] "RemoveContainer" containerID="64fe2efc9b0cf9f90ec24c3f31585fa8fa013a95a5be2e441dc3df803b3be81a"
Nov 03 12:30:02.745245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.745226    4546 generic.go:356] "Generic (PLEG): container finished" podID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerID="73c54626b383955e7844eded0b7582393895e79e9a69072539bc841133d1991d" exitCode=60
Nov 03 12:30:02.745360 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.745307    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" event={"ID":"3b0a4c6d-3f96-4746-ad49-2fb1981a623e","Type":"ContainerDied","Data":"73c54626b383955e7844eded0b7582393895e79e9a69072539bc841133d1991d"}
Nov 03 12:30:02.745471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.745451    4546 scope.go:122] "RemoveContainer" containerID="73c54626b383955e7844eded0b7582393895e79e9a69072539bc841133d1991d"
Nov 03 12:30:02.746883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.746864    4546 generic.go:356] "Generic (PLEG): container finished" podID="332bd850-8555-4835-84c5-1bc013fa34ff" containerID="db5ec10564fd905e3ad5dd3df0ad7126fae9514d1320902e4648ca7658cd8644" exitCode=60
Nov 03 12:30:02.746968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.746949    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" event={"ID":"332bd850-8555-4835-84c5-1bc013fa34ff","Type":"ContainerDied","Data":"db5ec10564fd905e3ad5dd3df0ad7126fae9514d1320902e4648ca7658cd8644"}
Nov 03 12:30:02.747128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.747114    4546 scope.go:122] "RemoveContainer" containerID="db5ec10564fd905e3ad5dd3df0ad7126fae9514d1320902e4648ca7658cd8644"
Nov 03 12:30:02.748569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.748551    4546 generic.go:356] "Generic (PLEG): container finished" podID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerID="90a86bde6612d87dfd832c5016d3d2f85721395a53a698c9e533bd4032c13c63" exitCode=60
Nov 03 12:30:02.748665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.748626    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" event={"ID":"7287e5ec-18c1-4d81-b3d9-44939c635b39","Type":"ContainerDied","Data":"90a86bde6612d87dfd832c5016d3d2f85721395a53a698c9e533bd4032c13c63"}
Nov 03 12:30:02.748875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.748848    4546 scope.go:122] "RemoveContainer" containerID="90a86bde6612d87dfd832c5016d3d2f85721395a53a698c9e533bd4032c13c63"
Nov 03 12:30:02.750206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.750185    4546 generic.go:356] "Generic (PLEG): container finished" podID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerID="a98400c2889616cff61794eeba5bcef09dad0d846f521b8cf9a0744de8010e83" exitCode=60
Nov 03 12:30:02.750297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.750230    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerDied","Data":"a98400c2889616cff61794eeba5bcef09dad0d846f521b8cf9a0744de8010e83"}
Nov 03 12:30:02.750424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:02.750412    4546 scope.go:122] "RemoveContainer" containerID="a98400c2889616cff61794eeba5bcef09dad0d846f521b8cf9a0744de8010e83"
Nov 03 12:30:03.662132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.662038    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 12:30:03.662132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.662092    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 12:30:03.689110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.689082    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"]
Nov 03 12:30:03.689358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.689334    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy" containerID="cri-o://b8d08c0cc2f30c03e0f019ca59ed8d38da6ef2d1509d786024b248bf3c28980f" gracePeriod=30
Nov 03 12:30:03.760788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.760719    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" event={"ID":"aec63eb2-c4da-4174-9ed8-9bb86d19b60d","Type":"ContainerStarted","Data":"d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b"}
Nov 03 12:30:03.763266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.763234    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" event={"ID":"cbd1cde6-9c5d-42b2-99b5-43daad17bcff","Type":"ContainerStarted","Data":"c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a"}
Nov 03 12:30:03.765343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.765306    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerStarted","Data":"660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca"}
Nov 03 12:30:03.767307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.767284    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" event={"ID":"3b0a4c6d-3f96-4746-ad49-2fb1981a623e","Type":"ContainerStarted","Data":"78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39"}
Nov 03 12:30:03.769386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.769351    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" event={"ID":"332bd850-8555-4835-84c5-1bc013fa34ff","Type":"ContainerStarted","Data":"0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09"}
Nov 03 12:30:03.771281 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.771261    4546 generic.go:356] "Generic (PLEG): container finished" podID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerID="e2be5c3fe511e77d1b30e37e986a5ad06b1a846afaf6da74bd93cb3a0ab7112c" exitCode=60
Nov 03 12:30:03.771379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.771338    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" event={"ID":"7287e5ec-18c1-4d81-b3d9-44939c635b39","Type":"ContainerDied","Data":"e2be5c3fe511e77d1b30e37e986a5ad06b1a846afaf6da74bd93cb3a0ab7112c"}
Nov 03 12:30:03.771430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.771386    4546 scope.go:122] "RemoveContainer" containerID="90a86bde6612d87dfd832c5016d3d2f85721395a53a698c9e533bd4032c13c63"
Nov 03 12:30:03.771597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.771584    4546 scope.go:122] "RemoveContainer" containerID="e2be5c3fe511e77d1b30e37e986a5ad06b1a846afaf6da74bd93cb3a0ab7112c"
Nov 03 12:30:03.771887 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:03.771865    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6_odh-ai-gateway-infra(7287e5ec-18c1-4d81-b3d9-44939c635b39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39"
Nov 03 12:30:03.773430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:03.773410    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerStarted","Data":"7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b"}
Nov 03 12:30:04.012472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.012428    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" podStartSLOduration=4.012412431 podStartE2EDuration="4.012412431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:04.010447683 +0000 UTC m=+2345.582031621" watchObservedRunningTime="2026-11-03 12:30:04.012412431 +0000 UTC m=+2345.583996345"
Nov 03 12:30:04.223815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.223735    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" podStartSLOduration=4.223716025 podStartE2EDuration="4.223716025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:04.222015385 +0000 UTC m=+2345.793599316" watchObservedRunningTime="2026-11-03 12:30:04.223716025 +0000 UTC m=+2345.795299939"
Nov 03 12:30:04.329769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.329695    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" podStartSLOduration=4.329682156 podStartE2EDuration="4.329682156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:04.326996487 +0000 UTC m=+2345.898580417" watchObservedRunningTime="2026-11-03 12:30:04.329682156 +0000 UTC m=+2345.901266068"
Nov 03 12:30:04.436881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.436778    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" podStartSLOduration=4.436763223 podStartE2EDuration="4.436763223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:04.434316913 +0000 UTC m=+2346.005900826" watchObservedRunningTime="2026-11-03 12:30:04.436763223 +0000 UTC m=+2346.008347126"
Nov 03 12:30:04.665644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.665616    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:30:04.666073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.666045    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:30:04.666669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.666653    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:30:04.784483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.784454    4546 generic.go:356] "Generic (PLEG): container finished" podID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerID="d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b" exitCode=60
Nov 03 12:30:04.784940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.784526    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" event={"ID":"aec63eb2-c4da-4174-9ed8-9bb86d19b60d","Type":"ContainerDied","Data":"d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b"}
Nov 03 12:30:04.784940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.784588    4546 scope.go:122] "RemoveContainer" containerID="486abbaba814744b7ee0096428ab02842ad981508164db2d9bd5b516b3c768d1"
Nov 03 12:30:04.784940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.784808    4546 scope.go:122] "RemoveContainer" containerID="d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b"
Nov 03 12:30:04.785115 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.785091    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6_odh-ai-gateway-infra(aec63eb2-c4da-4174-9ed8-9bb86d19b60d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d"
Nov 03 12:30:04.786873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.786853    4546 generic.go:356] "Generic (PLEG): container finished" podID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerID="c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a" exitCode=60
Nov 03 12:30:04.786993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.786930    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" event={"ID":"cbd1cde6-9c5d-42b2-99b5-43daad17bcff","Type":"ContainerDied","Data":"c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a"}
Nov 03 12:30:04.787122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.787107    4546 scope.go:122] "RemoveContainer" containerID="c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a"
Nov 03 12:30:04.787370 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.787349    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr_odh-ai-gateway-infra(cbd1cde6-9c5d-42b2-99b5-43daad17bcff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff"
Nov 03 12:30:04.788955 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.788934    4546 generic.go:356] "Generic (PLEG): container finished" podID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerID="660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca" exitCode=60
Nov 03 12:30:04.789080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.789043    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerDied","Data":"660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca"}
Nov 03 12:30:04.789210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.789193    4546 scope.go:122] "RemoveContainer" containerID="660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca"
Nov 03 12:30:04.789442 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.789410    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w_odh-ai-gateway-infra(4adc3273-1a5e-4ebc-94a0-c1dd4331d401)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401"
Nov 03 12:30:04.791224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.791202    4546 generic.go:356] "Generic (PLEG): container finished" podID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerID="78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39" exitCode=60
Nov 03 12:30:04.791329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.791250    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" event={"ID":"3b0a4c6d-3f96-4746-ad49-2fb1981a623e","Type":"ContainerDied","Data":"78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39"}
Nov 03 12:30:04.791503 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.791488    4546 scope.go:122] "RemoveContainer" containerID="78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39"
Nov 03 12:30:04.791740 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.791719    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q_odh-ai-gateway-infra(3b0a4c6d-3f96-4746-ad49-2fb1981a623e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e"
Nov 03 12:30:04.793190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.793168    4546 generic.go:356] "Generic (PLEG): container finished" podID="332bd850-8555-4835-84c5-1bc013fa34ff" containerID="0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09" exitCode=60
Nov 03 12:30:04.793280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.793236    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" event={"ID":"332bd850-8555-4835-84c5-1bc013fa34ff","Type":"ContainerDied","Data":"0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09"}
Nov 03 12:30:04.793488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.793460    4546 scope.go:122] "RemoveContainer" containerID="0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09"
Nov 03 12:30:04.793685 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.793658    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj_odh-ai-gateway-infra(332bd850-8555-4835-84c5-1bc013fa34ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" podUID="332bd850-8555-4835-84c5-1bc013fa34ff"
Nov 03 12:30:04.794980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.794963    4546 scope.go:122] "RemoveContainer" containerID="e2be5c3fe511e77d1b30e37e986a5ad06b1a846afaf6da74bd93cb3a0ab7112c"
Nov 03 12:30:04.795187 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.795165    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6_odh-ai-gateway-infra(7287e5ec-18c1-4d81-b3d9-44939c635b39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39"
Nov 03 12:30:04.796662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.796644    4546 generic.go:356] "Generic (PLEG): container finished" podID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerID="7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b" exitCode=60
Nov 03 12:30:04.796764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.796727    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerDied","Data":"7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b"}
Nov 03 12:30:04.796911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.796896    4546 scope.go:122] "RemoveContainer" containerID="7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b"
Nov 03 12:30:04.797117 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:04.797098    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht_odh-ai-gateway-infra(33258f10-6d09-46aa-9b61-f928198a9e6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e"
Nov 03 12:30:04.798722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.798699    4546 scope.go:122] "RemoveContainer" containerID="3093933905d3788fce7456adbf7068e02309d9ef6d310e748645876f0f23f319"
Nov 03 12:30:04.811789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.811771    4546 scope.go:122] "RemoveContainer" containerID="64fe2efc9b0cf9f90ec24c3f31585fa8fa013a95a5be2e441dc3df803b3be81a"
Nov 03 12:30:04.824942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.824916    4546 scope.go:122] "RemoveContainer" containerID="73c54626b383955e7844eded0b7582393895e79e9a69072539bc841133d1991d"
Nov 03 12:30:04.838102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.838082    4546 scope.go:122] "RemoveContainer" containerID="db5ec10564fd905e3ad5dd3df0ad7126fae9514d1320902e4648ca7658cd8644"
Nov 03 12:30:04.850822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:04.850799    4546 scope.go:122] "RemoveContainer" containerID="a98400c2889616cff61794eeba5bcef09dad0d846f521b8cf9a0744de8010e83"
Nov 03 12:30:05.805629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:05.805592    4546 scope.go:122] "RemoveContainer" containerID="78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39"
Nov 03 12:30:05.806165 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:05.805927    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q_odh-ai-gateway-infra(3b0a4c6d-3f96-4746-ad49-2fb1981a623e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e"
Nov 03 12:30:05.807346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:05.807324    4546 scope.go:122] "RemoveContainer" containerID="0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09"
Nov 03 12:30:05.807599 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:05.807576    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj_odh-ai-gateway-infra(332bd850-8555-4835-84c5-1bc013fa34ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" podUID="332bd850-8555-4835-84c5-1bc013fa34ff"
Nov 03 12:30:05.808948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:05.808928    4546 scope.go:122] "RemoveContainer" containerID="7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b"
Nov 03 12:30:05.809174 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:05.809154    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht_odh-ai-gateway-infra(33258f10-6d09-46aa-9b61-f928198a9e6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e"
Nov 03 12:30:05.810603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:05.810585    4546 scope.go:122] "RemoveContainer" containerID="d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b"
Nov 03 12:30:05.810829 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:05.810804    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6_odh-ai-gateway-infra(aec63eb2-c4da-4174-9ed8-9bb86d19b60d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d"
Nov 03 12:30:05.812182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:05.812165    4546 scope.go:122] "RemoveContainer" containerID="c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a"
Nov 03 12:30:05.812379 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:05.812357    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr_odh-ai-gateway-infra(cbd1cde6-9c5d-42b2-99b5-43daad17bcff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff"
Nov 03 12:30:05.813890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:05.813866    4546 scope.go:122] "RemoveContainer" containerID="660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca"
Nov 03 12:30:05.814125 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:05.814107    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w_odh-ai-gateway-infra(4adc3273-1a5e-4ebc-94a0-c1dd4331d401)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401"
Nov 03 12:30:07.077046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:07.077004    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"]
Nov 03 12:30:07.077648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:07.077313    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy" containerID="cri-o://2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c" gracePeriod=30
Nov 03 12:30:07.652533 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:07.652496    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:30:08.836284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:08.836251    4546 generic.go:356] "Generic (PLEG): container finished" podID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerID="b8d08c0cc2f30c03e0f019ca59ed8d38da6ef2d1509d786024b248bf3c28980f" exitCode=0
Nov 03 12:30:08.836736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:08.836318    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" event={"ID":"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a","Type":"ContainerDied","Data":"b8d08c0cc2f30c03e0f019ca59ed8d38da6ef2d1509d786024b248bf3c28980f"}
Nov 03 12:30:08.836736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:08.836392    4546 scope.go:122] "RemoveContainer" containerID="ec07f906fefaa8b85ba5f4d7d7da39f074e623fc09ed9777876c609f9f17d74e"
Nov 03 12:30:08.969555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:08.969533    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:30:09.059258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059227    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059281    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059382    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059425    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059558    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059661    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059670    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:09.059778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059682    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:09.059778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059689    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:09.059778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059712    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059806    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059836    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs\") pod \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\" (UID: \"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a\") "
Nov 03 12:30:09.059977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.059943    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:09.060244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.060223    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:09.060657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.060624    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.060657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.060652    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.060856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.060669    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.060856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.060684    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.060856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.060698    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.061990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.061966    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:09.062088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.062060    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:09.062298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.062269    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "kube-api-access-kscs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:09.062298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.062285    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy" pod "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" (UID: "faf6a433-0fa7-4c66-8bc2-30b0c6dd801a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:09.162239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.162201    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscs6\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-kube-api-access-kscs6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.162239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.162235    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.162464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.162251    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.162464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.162266    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:09.845084 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.845057    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"
Nov 03 12:30:09.845084 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.845065    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp" event={"ID":"faf6a433-0fa7-4c66-8bc2-30b0c6dd801a","Type":"ContainerDied","Data":"d4d701afcc35d1bf8efbd06f549b86e8aea6cffcf017f85c0f3a11910bea9bda"}
Nov 03 12:30:09.845638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:09.845138    4546 scope.go:122] "RemoveContainer" containerID="b8d08c0cc2f30c03e0f019ca59ed8d38da6ef2d1509d786024b248bf3c28980f"
Nov 03 12:30:10.000057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:10.000022    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"]
Nov 03 12:30:10.045665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:10.045628    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6db28-openshift-default-75fddd4f8f-nm7lp"]
Nov 03 12:30:10.574494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:10.574462    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" path="/var/lib/kubelet/pods/faf6a433-0fa7-4c66-8bc2-30b0c6dd801a/volumes"
Nov 03 12:30:12.355982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.355913    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:30:12.403343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403307    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403378    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403408    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403431    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403493    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403543    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403647    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403680    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:12.403894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403796    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.403946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403793    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:12.403946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403697    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:12.404045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403842    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:12.404045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.403995    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo\") pod \"2575e087-47b8-4c58-8702-686f6de7673e\" (UID: \"2575e087-47b8-4c58-8702-686f6de7673e\") "
Nov 03 12:30:12.404292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.404246    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:12.404557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.404536    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.404634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.404563    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.404634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.404576    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.404634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.404588    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.404634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.404602    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2575e087-47b8-4c58-8702-686f6de7673e-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.406060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.406038    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:12.406494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.406470    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:12.406586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.406533    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:12.406586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.406563    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts" pod "2575e087-47b8-4c58-8702-686f6de7673e" (UID: "2575e087-47b8-4c58-8702-686f6de7673e"). InnerVolumeSpecName "kube-api-access-j67ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:12.505722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.505684    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.505722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.505715    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2575e087-47b8-4c58-8702-686f6de7673e-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.505722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.505727    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2575e087-47b8-4c58-8702-686f6de7673e-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.506010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.505736    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67ts\" (UniqueName: \"kubernetes.io/projected/2575e087-47b8-4c58-8702-686f6de7673e-kube-api-access-j67ts\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:12.867500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.867466    4546 generic.go:356] "Generic (PLEG): container finished" podID="2575e087-47b8-4c58-8702-686f6de7673e" containerID="2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c" exitCode=0
Nov 03 12:30:12.867711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.867523    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"
Nov 03 12:30:12.867711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.867545    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" event={"ID":"2575e087-47b8-4c58-8702-686f6de7673e","Type":"ContainerDied","Data":"2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c"}
Nov 03 12:30:12.867711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.867606    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv" event={"ID":"2575e087-47b8-4c58-8702-686f6de7673e","Type":"ContainerDied","Data":"5dc8cd49b11b6645f4dcbc6039e9ce9ef697bfd8314759a332e5115ba010cc13"}
Nov 03 12:30:12.867711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.867633    4546 scope.go:122] "RemoveContainer" containerID="2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c"
Nov 03 12:30:12.877624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.877609    4546 scope.go:122] "RemoveContainer" containerID="5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"
Nov 03 12:30:12.886453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.886437    4546 scope.go:122] "RemoveContainer" containerID="2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c"
Nov 03 12:30:12.886712 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:12.886695    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c\": container with ID starting with 2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c not found: ID does not exist" containerID="2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c"
Nov 03 12:30:12.886807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.886720    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c"} err="failed to get container status \"2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c\": rpc error: code = NotFound desc = could not find container \"2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c\": container with ID starting with 2ebf0d600bdcb55e14405b624653c0a2ce390c75aa97774e0b9603c53754813c not found: ID does not exist"
Nov 03 12:30:12.886807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.886735    4546 scope.go:122] "RemoveContainer" containerID="5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"
Nov 03 12:30:12.887017 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:12.887004    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6\": container with ID starting with 5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6 not found: ID does not exist" containerID="5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"
Nov 03 12:30:12.887058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:12.887031    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6"} err="failed to get container status \"5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6\": rpc error: code = NotFound desc = could not find container \"5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6\": container with ID starting with 5d9215cfb40d2ea75431471c16b6caa3bc72aadeec577fb1e53a09d1f0f4c8c6 not found: ID does not exist"
Nov 03 12:30:13.021490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:13.021449    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"]
Nov 03 12:30:13.067393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:13.067362    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-607dc3-openshift-default-b9665bd77-6l4vv"]
Nov 03 12:30:13.609627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:13.609593    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"]
Nov 03 12:30:13.706803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:13.706774    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"]
Nov 03 12:30:14.566492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:14.566463    4546 scope.go:122] "RemoveContainer" containerID="660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca"
Nov 03 12:30:14.566689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:14.566547    4546 scope.go:122] "RemoveContainer" containerID="78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39"
Nov 03 12:30:14.587419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:14.587386    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2575e087-47b8-4c58-8702-686f6de7673e" path="/var/lib/kubelet/pods/2575e087-47b8-4c58-8702-686f6de7673e/volumes"
Nov 03 12:30:15.565691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.565661    4546 scope.go:122] "RemoveContainer" containerID="d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b"
Nov 03 12:30:15.566177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.566009    4546 scope.go:122] "RemoveContainer" containerID="0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09"
Nov 03 12:30:15.891734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.891708    4546 generic.go:356] "Generic (PLEG): container finished" podID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerID="63016a876cc9e24d8970b775247a262f66acf171f0e85d5ee3f7d879de3e5c92" exitCode=60
Nov 03 12:30:15.891940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.891795    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerDied","Data":"63016a876cc9e24d8970b775247a262f66acf171f0e85d5ee3f7d879de3e5c92"}
Nov 03 12:30:15.891940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.891860    4546 scope.go:122] "RemoveContainer" containerID="660fe34d8ce5efaf0354ccc402f8be60aa0ccaf32105dc209b8889951c7ec6ca"
Nov 03 12:30:15.892304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.892281    4546 scope.go:122] "RemoveContainer" containerID="63016a876cc9e24d8970b775247a262f66acf171f0e85d5ee3f7d879de3e5c92"
Nov 03 12:30:15.892688 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:15.892649    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w_odh-ai-gateway-infra(4adc3273-1a5e-4ebc-94a0-c1dd4331d401)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401"
Nov 03 12:30:15.893820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.893802    4546 generic.go:356] "Generic (PLEG): container finished" podID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerID="7fd4929df124357465eb69091456c45462906c7b072904808da9ed6230c281e9" exitCode=60
Nov 03 12:30:15.893910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.893847    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" event={"ID":"3b0a4c6d-3f96-4746-ad49-2fb1981a623e","Type":"ContainerDied","Data":"7fd4929df124357465eb69091456c45462906c7b072904808da9ed6230c281e9"}
Nov 03 12:30:15.894201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.894188    4546 scope.go:122] "RemoveContainer" containerID="7fd4929df124357465eb69091456c45462906c7b072904808da9ed6230c281e9"
Nov 03 12:30:15.894481 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:15.894465    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q_odh-ai-gateway-infra(3b0a4c6d-3f96-4746-ad49-2fb1981a623e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e"
Nov 03 12:30:15.905062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:15.905041    4546 scope.go:122] "RemoveContainer" containerID="78b83320d65407853c04b3355272f5a0aeeed87b85b15cc75b8ee766a43fce39"
Nov 03 12:30:16.906325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.906235    4546 generic.go:356] "Generic (PLEG): container finished" podID="332bd850-8555-4835-84c5-1bc013fa34ff" containerID="681fd566029f5417636b2ec14464278fb89abe435377534fd61dceb540ef798e" exitCode=60
Nov 03 12:30:16.906325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.906307    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" event={"ID":"332bd850-8555-4835-84c5-1bc013fa34ff","Type":"ContainerDied","Data":"681fd566029f5417636b2ec14464278fb89abe435377534fd61dceb540ef798e"}
Nov 03 12:30:16.906895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.906367    4546 scope.go:122] "RemoveContainer" containerID="0828b5c61031961e2a313dab1364249d0e3f211bb882688765f89d97881bce09"
Nov 03 12:30:16.906895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.906860    4546 scope.go:122] "RemoveContainer" containerID="681fd566029f5417636b2ec14464278fb89abe435377534fd61dceb540ef798e"
Nov 03 12:30:16.907209 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:16.907189    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj_odh-ai-gateway-infra(332bd850-8555-4835-84c5-1bc013fa34ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" podUID="332bd850-8555-4835-84c5-1bc013fa34ff"
Nov 03 12:30:16.908856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.908837    4546 generic.go:356] "Generic (PLEG): container finished" podID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerID="430afd1814bb7ce98505f94cac5f9a3787ecbc1df3d4e87ea6d4839477d43985" exitCode=60
Nov 03 12:30:16.908958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.908873    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" event={"ID":"aec63eb2-c4da-4174-9ed8-9bb86d19b60d","Type":"ContainerDied","Data":"430afd1814bb7ce98505f94cac5f9a3787ecbc1df3d4e87ea6d4839477d43985"}
Nov 03 12:30:16.909276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.909253    4546 scope.go:122] "RemoveContainer" containerID="430afd1814bb7ce98505f94cac5f9a3787ecbc1df3d4e87ea6d4839477d43985"
Nov 03 12:30:16.909617 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:16.909579    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6_odh-ai-gateway-infra(aec63eb2-c4da-4174-9ed8-9bb86d19b60d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d"
Nov 03 12:30:16.922216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:16.922196    4546 scope.go:122] "RemoveContainer" containerID="d0cb02e988bf9aebd65be17b173572a387c88f239754ae86d276026ee15fc47b"
Nov 03 12:30:17.012030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.011977    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"]
Nov 03 12:30:17.114472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.114439    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"]
Nov 03 12:30:17.284267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.284234    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:17.379378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.379351    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:17.381164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.381141    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q\" (UniqueName: \"kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q\") pod \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") "
Nov 03 12:30:17.381302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.381222    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca\") pod \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\" (UID: \"4adc3273-1a5e-4ebc-94a0-c1dd4331d401\") "
Nov 03 12:30:17.381671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.381638    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca" pod "4adc3273-1a5e-4ebc-94a0-c1dd4331d401" (UID: "4adc3273-1a5e-4ebc-94a0-c1dd4331d401"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:17.381845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.381824    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:17.383572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.383542    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q" pod "4adc3273-1a5e-4ebc-94a0-c1dd4331d401" (UID: "4adc3273-1a5e-4ebc-94a0-c1dd4331d401"). InnerVolumeSpecName "kube-api-access-ccr9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:17.484164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.484050    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca\") pod \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") "
Nov 03 12:30:17.484591 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.484548    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr\" (UniqueName: \"kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr\") pod \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\" (UID: \"3b0a4c6d-3f96-4746-ad49-2fb1981a623e\") "
Nov 03 12:30:17.485571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.484592    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca" pod "3b0a4c6d-3f96-4746-ad49-2fb1981a623e" (UID: "3b0a4c6d-3f96-4746-ad49-2fb1981a623e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:17.489620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.489591    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccr9q\" (UniqueName: \"kubernetes.io/projected/4adc3273-1a5e-4ebc-94a0-c1dd4331d401-kube-api-access-ccr9q\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:17.489764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.489623    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:17.491598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.491564    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr" pod "3b0a4c6d-3f96-4746-ad49-2fb1981a623e" (UID: "3b0a4c6d-3f96-4746-ad49-2fb1981a623e"). InnerVolumeSpecName "kube-api-access-q47mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:17.567212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.567181    4546 scope.go:122] "RemoveContainer" containerID="c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a"
Nov 03 12:30:17.567377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.567328    4546 scope.go:122] "RemoveContainer" containerID="e2be5c3fe511e77d1b30e37e986a5ad06b1a846afaf6da74bd93cb3a0ab7112c"
Nov 03 12:30:17.591539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.591508    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47mr\" (UniqueName: \"kubernetes.io/projected/3b0a4c6d-3f96-4746-ad49-2fb1981a623e-kube-api-access-q47mr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:17.929225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.929189    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"
Nov 03 12:30:17.934772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.931259    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"
Nov 03 12:30:17.942662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.942624    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w" event={"ID":"4adc3273-1a5e-4ebc-94a0-c1dd4331d401","Type":"ContainerDied","Data":"7c725926486182a6cd796057907d8dc4992b6f84b9a1f7304fb8884e446adebe"}
Nov 03 12:30:17.942975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.942954    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q" event={"ID":"3b0a4c6d-3f96-4746-ad49-2fb1981a623e","Type":"ContainerDied","Data":"054da918d82ea5bf79c4b8b9b8f4b6c6fe4392f31c23887cd40f3ce56b20aabc"}
Nov 03 12:30:17.943160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.943103    4546 scope.go:122] "RemoveContainer" containerID="63016a876cc9e24d8970b775247a262f66acf171f0e85d5ee3f7d879de3e5c92"
Nov 03 12:30:17.965251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:17.965227    4546 scope.go:122] "RemoveContainer" containerID="7fd4929df124357465eb69091456c45462906c7b072904808da9ed6230c281e9"
Nov 03 12:30:18.029730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.029690    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"]
Nov 03 12:30:18.089438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.089208    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"]
Nov 03 12:30:18.093182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.093154    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895150-7d58w"]
Nov 03 12:30:18.162776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.136512    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"]
Nov 03 12:30:18.206489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.200460    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"]
Nov 03 12:30:18.206489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.205802    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895150-vfx4q"]
Nov 03 12:30:18.394203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.394177    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.422518    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"]
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423564    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423585    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423604    4546 cpu_manager.go:414] "containerMap: removing container" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423612    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423623    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423630    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423648    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423656    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423667    4546 cpu_manager.go:414] "containerMap: removing container" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423675    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423696    4546 cpu_manager.go:414] "containerMap: removing container" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423704    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423716    4546 cpu_manager.go:414] "containerMap: removing container" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423723    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423734    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423740    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423773    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423780    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423800    4546 cpu_manager.go:414] "containerMap: removing container" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423807    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423820    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423827    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423840    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.423848    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424008    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424021    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424031    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424049    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2575e087-47b8-4c58-8702-686f6de7673e" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424058    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424068    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424076    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf6a433-0fa7-4c66-8bc2-30b0c6dd801a" containerName="istio-proxy"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424086    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424094    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424105    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:18.426005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.424115    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" containerName="cleanup"
Nov 03 12:30:18.465358 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.462405    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.488768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.488663    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"]
Nov 03 12:30:18.504043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504011    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs\" (UniqueName: \"kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs\") pod \"332bd850-8555-4835-84c5-1bc013fa34ff\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") "
Nov 03 12:30:18.504240 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504219    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca\") pod \"332bd850-8555-4835-84c5-1bc013fa34ff\" (UID: \"332bd850-8555-4835-84c5-1bc013fa34ff\") "
Nov 03 12:30:18.504554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504533    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.504652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504629    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9rp\" (UniqueName: \"kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.504728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504710    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.504810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504736    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca" pod "332bd850-8555-4835-84c5-1bc013fa34ff" (UID: "332bd850-8555-4835-84c5-1bc013fa34ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:18.504916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.504903    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332bd850-8555-4835-84c5-1bc013fa34ff-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.506323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.506301    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs" pod "332bd850-8555-4835-84c5-1bc013fa34ff" (UID: "332bd850-8555-4835-84c5-1bc013fa34ff"). InnerVolumeSpecName "kube-api-access-l6lfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:18.533639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.532196    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:30:18.567775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.567719    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:18.568801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.568775    4546 scope.go:122] "RemoveContainer" containerID="7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b"
Nov 03 12:30:18.586345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.586320    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" path="/var/lib/kubelet/pods/3b0a4c6d-3f96-4746-ad49-2fb1981a623e/volumes"
Nov 03 12:30:18.586938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.586923    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" path="/var/lib/kubelet/pods/4adc3273-1a5e-4ebc-94a0-c1dd4331d401/volumes"
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.606609    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca\") pod \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") "
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.611087    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf" podUID="9795ff78-c344-4d5c-8d95-ac2466e1b4ed" containerName="payload-processing" containerID="cri-o://c66fc17e5d4009ef1533cf0f089901e9b7a744e8a4fa01f71afa5d142e71d2bd" gracePeriod=25
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.613579    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca" pod "aec63eb2-c4da-4174-9ed8-9bb86d19b60d" (UID: "aec63eb2-c4da-4174-9ed8-9bb86d19b60d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.630952    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x\" (UniqueName: \"kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x\") pod \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\" (UID: \"aec63eb2-c4da-4174-9ed8-9bb86d19b60d\") "
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.631282    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.631501    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.631616    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9rp\" (UniqueName: \"kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.631810    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6lfs\" (UniqueName: \"kubernetes.io/projected/332bd850-8555-4835-84c5-1bc013fa34ff-kube-api-access-l6lfs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.643833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.631830    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.648814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.648778    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.650417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.650390    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.664603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.664553    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x" pod "aec63eb2-c4da-4174-9ed8-9bb86d19b60d" (UID: "aec63eb2-c4da-4174-9ed8-9bb86d19b60d"). InnerVolumeSpecName "kube-api-access-h2r5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:18.701243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.701209    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9rp\" (UniqueName: \"kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp\") pod \"maas-api-revoke-sub-22192dcc-dt6lk\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.709927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.707363    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt" podUID="1b02228d-367c-4f89-b550-c16ebe85c372" containerName="payload-pre-processing" containerID="cri-o://dc366965ab6ba8f638d7ac6ca45b17584d8cf968015c6d9d91a69fe2d5881307" gracePeriod=25
Nov 03 12:30:18.732932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.732890    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2r5x\" (UniqueName: \"kubernetes.io/projected/aec63eb2-c4da-4174-9ed8-9bb86d19b60d-kube-api-access-h2r5x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.863479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.860644    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:18.943327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.943284    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj" event={"ID":"332bd850-8555-4835-84c5-1bc013fa34ff","Type":"ContainerDied","Data":"2cfe98b135c4d7bcff461848dae712fab56cff7cc63c62127de5cdb804809949"}
Nov 03 12:30:18.943905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.943374    4546 scope.go:122] "RemoveContainer" containerID="681fd566029f5417636b2ec14464278fb89abe435377534fd61dceb540ef798e"
Nov 03 12:30:18.943905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.943315    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"
Nov 03 12:30:18.951094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.949701    4546 generic.go:356] "Generic (PLEG): container finished" podID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerID="6d3db9c80827abf36078636ac65e20f197f7c3e963de72d8b701daea029d0487" exitCode=60
Nov 03 12:30:18.951094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.949799    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" event={"ID":"7287e5ec-18c1-4d81-b3d9-44939c635b39","Type":"ContainerDied","Data":"6d3db9c80827abf36078636ac65e20f197f7c3e963de72d8b701daea029d0487"}
Nov 03 12:30:18.951094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.950262    4546 scope.go:122] "RemoveContainer" containerID="6d3db9c80827abf36078636ac65e20f197f7c3e963de72d8b701daea029d0487"
Nov 03 12:30:18.951094 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:18.950645    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6_odh-ai-gateway-infra(7287e5ec-18c1-4d81-b3d9-44939c635b39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39"
Nov 03 12:30:18.952948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.952262    4546 generic.go:356] "Generic (PLEG): container finished" podID="9795ff78-c344-4d5c-8d95-ac2466e1b4ed" containerID="c66fc17e5d4009ef1533cf0f089901e9b7a744e8a4fa01f71afa5d142e71d2bd" exitCode=0
Nov 03 12:30:18.952948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.952500    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf" event={"ID":"9795ff78-c344-4d5c-8d95-ac2466e1b4ed","Type":"ContainerDied","Data":"c66fc17e5d4009ef1533cf0f089901e9b7a744e8a4fa01f71afa5d142e71d2bd"}
Nov 03 12:30:18.957307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.954563    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6" event={"ID":"aec63eb2-c4da-4174-9ed8-9bb86d19b60d","Type":"ContainerDied","Data":"283b66402d27b0af1686d03df99bcebe485746ad2a115934823aa9f0becb964c"}
Nov 03 12:30:18.957307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.954642    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"
Nov 03 12:30:18.961247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.961134    4546 scope.go:122] "RemoveContainer" containerID="e2be5c3fe511e77d1b30e37e986a5ad06b1a846afaf6da74bd93cb3a0ab7112c"
Nov 03 12:30:18.962789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.962709    4546 generic.go:356] "Generic (PLEG): container finished" podID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerID="3d91ad3394f90100757fa1aa405d2c7bcc6a4457eedeb43b131825ca1ff1ad83" exitCode=60
Nov 03 12:30:18.962889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.962797    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" event={"ID":"cbd1cde6-9c5d-42b2-99b5-43daad17bcff","Type":"ContainerDied","Data":"3d91ad3394f90100757fa1aa405d2c7bcc6a4457eedeb43b131825ca1ff1ad83"}
Nov 03 12:30:18.963544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.963202    4546 scope.go:122] "RemoveContainer" containerID="3d91ad3394f90100757fa1aa405d2c7bcc6a4457eedeb43b131825ca1ff1ad83"
Nov 03 12:30:18.963544 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:18.963491    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr_odh-ai-gateway-infra(cbd1cde6-9c5d-42b2-99b5-43daad17bcff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff"
Nov 03 12:30:18.966222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.966077    4546 generic.go:356] "Generic (PLEG): container finished" podID="1b02228d-367c-4f89-b550-c16ebe85c372" containerID="dc366965ab6ba8f638d7ac6ca45b17584d8cf968015c6d9d91a69fe2d5881307" exitCode=0
Nov 03 12:30:18.966222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.966161    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt" event={"ID":"1b02228d-367c-4f89-b550-c16ebe85c372","Type":"ContainerDied","Data":"dc366965ab6ba8f638d7ac6ca45b17584d8cf968015c6d9d91a69fe2d5881307"}
Nov 03 12:30:18.979876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.979854    4546 scope.go:122] "RemoveContainer" containerID="430afd1814bb7ce98505f94cac5f9a3787ecbc1df3d4e87ea6d4839477d43985"
Nov 03 12:30:18.996843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:18.996819    4546 scope.go:122] "RemoveContainer" containerID="c78c396c13f288a5cc4c8bb2ac977ce7d1c8c35309aae2e300c3f1a43d79e60a"
Nov 03 12:30:19.002459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.002434    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:30:19.061152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.061125    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:30:19.141044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.140964    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56\" (UniqueName: \"kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56\") pod \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") "
Nov 03 12:30:19.141216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141049    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume\") pod \"1b02228d-367c-4f89-b550-c16ebe85c372\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") "
Nov 03 12:30:19.141216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141107    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd\" (UniqueName: \"kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd\") pod \"1b02228d-367c-4f89-b550-c16ebe85c372\" (UID: \"1b02228d-367c-4f89-b550-c16ebe85c372\") "
Nov 03 12:30:19.141331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141211    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume\") pod \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\" (UID: \"9795ff78-c344-4d5c-8d95-ac2466e1b4ed\") "
Nov 03 12:30:19.141483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141461    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume" pod "1b02228d-367c-4f89-b550-c16ebe85c372" (UID: "1b02228d-367c-4f89-b550-c16ebe85c372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:19.141549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141507    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume" pod "9795ff78-c344-4d5c-8d95-ac2466e1b4ed" (UID: "9795ff78-c344-4d5c-8d95-ac2466e1b4ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:19.141718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141699    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02228d-367c-4f89-b550-c16ebe85c372-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:19.141781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.141721    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:19.143305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.143277    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56" pod "9795ff78-c344-4d5c-8d95-ac2466e1b4ed" (UID: "9795ff78-c344-4d5c-8d95-ac2466e1b4ed"). InnerVolumeSpecName "kube-api-access-fpk56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:19.143403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.143337    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd" pod "1b02228d-367c-4f89-b550-c16ebe85c372" (UID: "1b02228d-367c-4f89-b550-c16ebe85c372"). InnerVolumeSpecName "kube-api-access-8b9wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:19.205234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.205196    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"]
Nov 03 12:30:19.209293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.209261    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2f3eb2-29895150-bjzw6"]
Nov 03 12:30:19.243672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.243635    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9wd\" (UniqueName: \"kubernetes.io/projected/1b02228d-367c-4f89-b550-c16ebe85c372-kube-api-access-8b9wd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:19.243902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.243675    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpk56\" (UniqueName: \"kubernetes.io/projected/9795ff78-c344-4d5c-8d95-ac2466e1b4ed-kube-api-access-fpk56\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:19.247028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.247002    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"]
Nov 03 12:30:19.247970 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:19.247937    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cded93_9084_442c_8191_2a6781908c95.slice/crio-45454d488f5f2078fbf80cf844931e2314493119f98219e016e238a39d02bc02 WatchSource:0}: Error finding container 45454d488f5f2078fbf80cf844931e2314493119f98219e016e238a39d02bc02: Status 404 returned error can't find the container with id 45454d488f5f2078fbf80cf844931e2314493119f98219e016e238a39d02bc02
Nov 03 12:30:19.309708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.309673    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"]
Nov 03 12:30:19.313531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.313497    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895150-wjhhj"]
Nov 03 12:30:19.978718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.978680    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf" event={"ID":"9795ff78-c344-4d5c-8d95-ac2466e1b4ed","Type":"ContainerDied","Data":"756f32cc53ecdfe764acbb323cacfc9bf8271bf256ff9dbc0888cb3b9ae639f6"}
Nov 03 12:30:19.979301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.978742    4546 scope.go:122] "RemoveContainer" containerID="c66fc17e5d4009ef1533cf0f089901e9b7a744e8a4fa01f71afa5d142e71d2bd"
Nov 03 12:30:19.979301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.978899    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"
Nov 03 12:30:19.980971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.980947    4546 generic.go:356] "Generic (PLEG): container finished" podID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerID="9001a70a1bf2ddec020ff4415ac467f345ffb76237452897608c1b8f5e21f629" exitCode=60
Nov 03 12:30:19.981064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.981014    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerDied","Data":"9001a70a1bf2ddec020ff4415ac467f345ffb76237452897608c1b8f5e21f629"}
Nov 03 12:30:19.981414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.981392    4546 scope.go:122] "RemoveContainer" containerID="9001a70a1bf2ddec020ff4415ac467f345ffb76237452897608c1b8f5e21f629"
Nov 03 12:30:19.981717 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:19.981693    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht_odh-ai-gateway-infra(33258f10-6d09-46aa-9b61-f928198a9e6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e"
Nov 03 12:30:19.989346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.989320    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk" event={"ID":"d3cded93-9084-442c-8191-2a6781908c95","Type":"ContainerStarted","Data":"6a2422202fab60de2013cb9947d993b6acc67c02fa7a5239047d66f575358754"}
Nov 03 12:30:19.989461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.989390    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk" event={"ID":"d3cded93-9084-442c-8191-2a6781908c95","Type":"ContainerStarted","Data":"45454d488f5f2078fbf80cf844931e2314493119f98219e016e238a39d02bc02"}
Nov 03 12:30:19.989776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.989741    4546 scope.go:122] "RemoveContainer" containerID="7b49053454b73ee9ea8b086b7dbf72ee6bdcaaa1ca2ff9f5475f09336043049b"
Nov 03 12:30:19.990600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.990569    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt" event={"ID":"1b02228d-367c-4f89-b550-c16ebe85c372","Type":"ContainerDied","Data":"3da133f50272c752c50d4671e5853d60fc2d3f75f49af77c8b826327b03ba1f9"}
Nov 03 12:30:19.990672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:19.990602    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"
Nov 03 12:30:20.004958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.004931    4546 scope.go:122] "RemoveContainer" containerID="dc366965ab6ba8f638d7ac6ca45b17584d8cf968015c6d9d91a69fe2d5881307"
Nov 03 12:30:20.111564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.111533    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"]
Nov 03 12:30:20.133828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.133782    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"]
Nov 03 12:30:20.139700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.139664    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04d73d-7465c74d4f-pq9mf"]
Nov 03 12:30:20.243526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.243432    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk" podStartSLOduration=2.243418437 podStartE2EDuration="2.243418437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:20.243049177 +0000 UTC m=+2361.814633118" watchObservedRunningTime="2026-11-03 12:30:20.243418437 +0000 UTC m=+2361.815002350"
Nov 03 12:30:20.293117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.293090    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:20.357513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.357483    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp\" (UniqueName: \"kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp\") pod \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") "
Nov 03 12:30:20.357726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.357636    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca\") pod \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\" (UID: \"cbd1cde6-9c5d-42b2-99b5-43daad17bcff\") "
Nov 03 12:30:20.358034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.358009    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca" pod "cbd1cde6-9c5d-42b2-99b5-43daad17bcff" (UID: "cbd1cde6-9c5d-42b2-99b5-43daad17bcff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:20.359776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.359737    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp" pod "cbd1cde6-9c5d-42b2-99b5-43daad17bcff" (UID: "cbd1cde6-9c5d-42b2-99b5-43daad17bcff"). InnerVolumeSpecName "kube-api-access-n2ldp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:20.423834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.423794    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"]
Nov 03 12:30:20.452004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.451950    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"]
Nov 03 12:30:20.455665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.455640    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04d73d-56d9c95cf-lcrwt"]
Nov 03 12:30:20.459519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.459492    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:20.459620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.459520    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2ldp\" (UniqueName: \"kubernetes.io/projected/cbd1cde6-9c5d-42b2-99b5-43daad17bcff-kube-api-access-n2ldp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:20.576985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.576956    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b02228d-367c-4f89-b550-c16ebe85c372" path="/var/lib/kubelet/pods/1b02228d-367c-4f89-b550-c16ebe85c372/volumes"
Nov 03 12:30:20.577596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.577576    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332bd850-8555-4835-84c5-1bc013fa34ff" path="/var/lib/kubelet/pods/332bd850-8555-4835-84c5-1bc013fa34ff/volumes"
Nov 03 12:30:20.578181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.578163    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9795ff78-c344-4d5c-8d95-ac2466e1b4ed" path="/var/lib/kubelet/pods/9795ff78-c344-4d5c-8d95-ac2466e1b4ed/volumes"
Nov 03 12:30:20.579101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.579081    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" path="/var/lib/kubelet/pods/aec63eb2-c4da-4174-9ed8-9bb86d19b60d/volumes"
Nov 03 12:30:20.599734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.599700    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:20.661227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.661192    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78\" (UniqueName: \"kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78\") pod \"7287e5ec-18c1-4d81-b3d9-44939c635b39\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") "
Nov 03 12:30:20.661442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.661423    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca\") pod \"7287e5ec-18c1-4d81-b3d9-44939c635b39\" (UID: \"7287e5ec-18c1-4d81-b3d9-44939c635b39\") "
Nov 03 12:30:20.662082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.662049    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca" pod "7287e5ec-18c1-4d81-b3d9-44939c635b39" (UID: "7287e5ec-18c1-4d81-b3d9-44939c635b39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:20.663992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.663959    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78" pod "7287e5ec-18c1-4d81-b3d9-44939c635b39" (UID: "7287e5ec-18c1-4d81-b3d9-44939c635b39"). InnerVolumeSpecName "kube-api-access-c8h78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:20.763547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.763509    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7287e5ec-18c1-4d81-b3d9-44939c635b39-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:20.763547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.763538    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8h78\" (UniqueName: \"kubernetes.io/projected/7287e5ec-18c1-4d81-b3d9-44939c635b39-kube-api-access-c8h78\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:20.998641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.998609    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"
Nov 03 12:30:20.999162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.998608    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr" event={"ID":"cbd1cde6-9c5d-42b2-99b5-43daad17bcff","Type":"ContainerDied","Data":"110e5c6bb13f85cb09abdc94b8babe180cf9f4333f360fa9044e2e0744b837fe"}
Nov 03 12:30:20.999162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:20.998798    4546 scope.go:122] "RemoveContainer" containerID="3d91ad3394f90100757fa1aa405d2c7bcc6a4457eedeb43b131825ca1ff1ad83"
Nov 03 12:30:21.000640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.000616    4546 generic.go:356] "Generic (PLEG): container finished" podID="d3cded93-9084-442c-8191-2a6781908c95" containerID="6a2422202fab60de2013cb9947d993b6acc67c02fa7a5239047d66f575358754" exitCode=0
Nov 03 12:30:21.000776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.000700    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk" event={"ID":"d3cded93-9084-442c-8191-2a6781908c95","Type":"ContainerDied","Data":"6a2422202fab60de2013cb9947d993b6acc67c02fa7a5239047d66f575358754"}
Nov 03 12:30:21.003319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.003299    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"
Nov 03 12:30:21.003411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.003298    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6" event={"ID":"7287e5ec-18c1-4d81-b3d9-44939c635b39","Type":"ContainerDied","Data":"692a6084d9ee74650c918d6396e31958fbd3f920f27bbcc371b3e61b7e2169a4"}
Nov 03 12:30:21.009465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.009447    4546 scope.go:122] "RemoveContainer" containerID="6d3db9c80827abf36078636ac65e20f197f7c3e963de72d8b701daea029d0487"
Nov 03 12:30:21.151937 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.151890    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"]
Nov 03 12:30:21.156226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.156177    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04d73d-29895150-8wxjr"]
Nov 03 12:30:21.259234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.259136    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"]
Nov 03 12:30:21.262617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.262583    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c02ae4-29895150-k7nd6"]
Nov 03 12:30:21.358983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.358950    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"]
Nov 03 12:30:21.524791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.524702    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:21.674319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.674280    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766\" (UniqueName: \"kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766\") pod \"33258f10-6d09-46aa-9b61-f928198a9e6e\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") "
Nov 03 12:30:21.674542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.674376    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca\") pod \"33258f10-6d09-46aa-9b61-f928198a9e6e\" (UID: \"33258f10-6d09-46aa-9b61-f928198a9e6e\") "
Nov 03 12:30:21.674847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.674813    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca" pod "33258f10-6d09-46aa-9b61-f928198a9e6e" (UID: "33258f10-6d09-46aa-9b61-f928198a9e6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:21.675201 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.675175    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33258f10-6d09-46aa-9b61-f928198a9e6e-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:21.677346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.677315    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766" pod "33258f10-6d09-46aa-9b61-f928198a9e6e" (UID: "33258f10-6d09-46aa-9b61-f928198a9e6e"). InnerVolumeSpecName "kube-api-access-wt766". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:21.776622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:21.776548    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt766\" (UniqueName: \"kubernetes.io/projected/33258f10-6d09-46aa-9b61-f928198a9e6e-kube-api-access-wt766\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:22.015822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.015791    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"
Nov 03 12:30:22.015822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.015798    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht" event={"ID":"33258f10-6d09-46aa-9b61-f928198a9e6e","Type":"ContainerDied","Data":"2e6777912d244c4aacf1d411a9eadd448cd72dc513c48c5c7a1940923cebe981"}
Nov 03 12:30:22.016356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.015882    4546 scope.go:122] "RemoveContainer" containerID="9001a70a1bf2ddec020ff4415ac467f345ffb76237452897608c1b8f5e21f629"
Nov 03 12:30:22.172659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.172625    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"]
Nov 03 12:30:22.177868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.177835    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895150-vrxht"]
Nov 03 12:30:22.193039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.193016    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:22.283936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.283841    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca\") pod \"d3cded93-9084-442c-8191-2a6781908c95\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") "
Nov 03 12:30:22.284092 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.284001    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp\" (UniqueName: \"kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp\") pod \"d3cded93-9084-442c-8191-2a6781908c95\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") "
Nov 03 12:30:22.284167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.284115    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca\") pod \"d3cded93-9084-442c-8191-2a6781908c95\" (UID: \"d3cded93-9084-442c-8191-2a6781908c95\") "
Nov 03 12:30:22.284673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.284641    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca" pod "d3cded93-9084-442c-8191-2a6781908c95" (UID: "d3cded93-9084-442c-8191-2a6781908c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:22.287196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.287162    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp" pod "d3cded93-9084-442c-8191-2a6781908c95" (UID: "d3cded93-9084-442c-8191-2a6781908c95"). InnerVolumeSpecName "kube-api-access-jr9rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:22.324419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.316578    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca" pod "d3cded93-9084-442c-8191-2a6781908c95" (UID: "d3cded93-9084-442c-8191-2a6781908c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:22.385395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.385360    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9rp\" (UniqueName: \"kubernetes.io/projected/d3cded93-9084-442c-8191-2a6781908c95-kube-api-access-jr9rp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:22.385395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.385396    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cded93-9084-442c-8191-2a6781908c95-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:22.385593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.385411    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cded93-9084-442c-8191-2a6781908c95-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:22.574327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.574255    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" path="/var/lib/kubelet/pods/33258f10-6d09-46aa-9b61-f928198a9e6e/volumes"
Nov 03 12:30:22.574654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.574641    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" path="/var/lib/kubelet/pods/7287e5ec-18c1-4d81-b3d9-44939c635b39/volumes"
Nov 03 12:30:22.575043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.575030    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" path="/var/lib/kubelet/pods/cbd1cde6-9c5d-42b2-99b5-43daad17bcff/volumes"
Nov 03 12:30:22.726333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726294    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"]
Nov 03 12:30:22.726902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726886    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9795ff78-c344-4d5c-8d95-ac2466e1b4ed" containerName="payload-processing"
Nov 03 12:30:22.726902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726902    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9795ff78-c344-4d5c-8d95-ac2466e1b4ed" containerName="payload-processing"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726911    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726920    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726933    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726938    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726948    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d3cded93-9084-442c-8191-2a6781908c95" containerName="revoke-keys"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726953    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cded93-9084-442c-8191-2a6781908c95" containerName="revoke-keys"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726964    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726968    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726985    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726989    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.726997    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727001    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727009    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727012    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727020    4546 cpu_manager.go:414] "containerMap: removing container" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727024    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727036    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727040    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727049    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727053    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727066    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727069    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727082    4546 cpu_manager.go:414] "containerMap: removing container" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727086    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727093    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1b02228d-367c-4f89-b550-c16ebe85c372" containerName="payload-pre-processing"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727097    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b02228d-367c-4f89-b550-c16ebe85c372" containerName="payload-pre-processing"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727177    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727184    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adc3273-1a5e-4ebc-94a0-c1dd4331d401" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727193    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727198    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727203    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727207    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727213    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7287e5ec-18c1-4d81-b3d9-44939c635b39" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727220    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727224    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a4c6d-3f96-4746-ad49-2fb1981a623e" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727229    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727234    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b02228d-367c-4f89-b550-c16ebe85c372" containerName="payload-pre-processing"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727241    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9795ff78-c344-4d5c-8d95-ac2466e1b4ed" containerName="payload-processing"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727246    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:22.727611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.727250    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cded93-9084-442c-8191-2a6781908c95" containerName="revoke-keys"
Nov 03 12:30:22.731919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.731903    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.784515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.784484    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"]
Nov 03 12:30:22.890664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.890576    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.890882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.890714    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.890882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.890866    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6c2\" (UniqueName: \"kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.992475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.992436    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.992655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.992530    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.992655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.992609    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6c2\" (UniqueName: \"kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.993287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.993262    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:22.995343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:22.995317    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:23.025189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:23.025154    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk" event={"ID":"d3cded93-9084-442c-8191-2a6781908c95","Type":"ContainerDied","Data":"45454d488f5f2078fbf80cf844931e2314493119f98219e016e238a39d02bc02"}
Nov 03 12:30:23.025545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:23.025212    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45454d488f5f2078fbf80cf844931e2314493119f98219e016e238a39d02bc02"
Nov 03 12:30:23.025545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:23.025167    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"
Nov 03 12:30:23.045884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:23.045857    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6c2\" (UniqueName: \"kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2\") pod \"maas-api-revoke-sub-9e400204-cvp9s\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:23.345897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:23.345869    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:23.711714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:23.711681    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"]
Nov 03 12:30:23.712576 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:23.712549    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25784e49_ebd6_43c9_ad16_0c1d200ded42.slice/crio-bf31d4f6bafccd047fb08bc140d6c840abdc00f63339287f48845225b62e2176 WatchSource:0}: Error finding container bf31d4f6bafccd047fb08bc140d6c840abdc00f63339287f48845225b62e2176: Status 404 returned error can't find the container with id bf31d4f6bafccd047fb08bc140d6c840abdc00f63339287f48845225b62e2176
Nov 03 12:30:24.037531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:24.037493    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s" event={"ID":"25784e49-ebd6-43c9-ad16-0c1d200ded42","Type":"ContainerStarted","Data":"e8b12764b28d503791e3d2a11ea43649239275e01fbb32d69c09834de96c4b3b"}
Nov 03 12:30:24.037984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:24.037560    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s" event={"ID":"25784e49-ebd6-43c9-ad16-0c1d200ded42","Type":"ContainerStarted","Data":"bf31d4f6bafccd047fb08bc140d6c840abdc00f63339287f48845225b62e2176"}
Nov 03 12:30:24.143669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:24.143607    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s" podStartSLOduration=2.143590702 podStartE2EDuration="2.143590702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:24.141892348 +0000 UTC m=+2365.713476260" watchObservedRunningTime="2026-11-03 12:30:24.143590702 +0000 UTC m=+2365.715174615"
Nov 03 12:30:25.045703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:25.045670    4546 generic.go:356] "Generic (PLEG): container finished" podID="25784e49-ebd6-43c9-ad16-0c1d200ded42" containerID="e8b12764b28d503791e3d2a11ea43649239275e01fbb32d69c09834de96c4b3b" exitCode=0
Nov 03 12:30:25.046166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:25.045771    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s" event={"ID":"25784e49-ebd6-43c9-ad16-0c1d200ded42","Type":"ContainerDied","Data":"e8b12764b28d503791e3d2a11ea43649239275e01fbb32d69c09834de96c4b3b"}
Nov 03 12:30:26.219727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.219700    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:26.335321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.335228    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2\" (UniqueName: \"kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2\") pod \"25784e49-ebd6-43c9-ad16-0c1d200ded42\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") "
Nov 03 12:30:26.335512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.335393    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca\") pod \"25784e49-ebd6-43c9-ad16-0c1d200ded42\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") "
Nov 03 12:30:26.335512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.335434    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca\") pod \"25784e49-ebd6-43c9-ad16-0c1d200ded42\" (UID: \"25784e49-ebd6-43c9-ad16-0c1d200ded42\") "
Nov 03 12:30:26.335840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.335816    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca" pod "25784e49-ebd6-43c9-ad16-0c1d200ded42" (UID: "25784e49-ebd6-43c9-ad16-0c1d200ded42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:26.337870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.337840    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2" pod "25784e49-ebd6-43c9-ad16-0c1d200ded42" (UID: "25784e49-ebd6-43c9-ad16-0c1d200ded42"). InnerVolumeSpecName "kube-api-access-8l6c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:26.338038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.337931    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca" pod "25784e49-ebd6-43c9-ad16-0c1d200ded42" (UID: "25784e49-ebd6-43c9-ad16-0c1d200ded42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:26.437478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.437436    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6c2\" (UniqueName: \"kubernetes.io/projected/25784e49-ebd6-43c9-ad16-0c1d200ded42-kube-api-access-8l6c2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:26.437478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.437471    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:26.437478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:26.437480    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25784e49-ebd6-43c9-ad16-0c1d200ded42-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:27.060954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:27.060914    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"
Nov 03 12:30:27.061134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:27.060952    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s" event={"ID":"25784e49-ebd6-43c9-ad16-0c1d200ded42","Type":"ContainerDied","Data":"bf31d4f6bafccd047fb08bc140d6c840abdc00f63339287f48845225b62e2176"}
Nov 03 12:30:27.061134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:27.061002    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf31d4f6bafccd047fb08bc140d6c840abdc00f63339287f48845225b62e2176"
Nov 03 12:30:32.769078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769044    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"]
Nov 03 12:30:32.769678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769661    4546 cpu_manager.go:414] "containerMap: removing container" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:32.769727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769680    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:32.769727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769719    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25784e49-ebd6-43c9-ad16-0c1d200ded42" containerName="revoke-keys"
Nov 03 12:30:32.769727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769725    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25784e49-ebd6-43c9-ad16-0c1d200ded42" containerName="revoke-keys"
Nov 03 12:30:32.769855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769736    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:32.769855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769741    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:32.769855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769836    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33258f10-6d09-46aa-9b61-f928198a9e6e" containerName="cleanup"
Nov 03 12:30:32.769855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769845    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd1cde6-9c5d-42b2-99b5-43daad17bcff" containerName="cleanup"
Nov 03 12:30:32.769855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769852    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25784e49-ebd6-43c9-ad16-0c1d200ded42" containerName="revoke-keys"
Nov 03 12:30:32.770045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.769860    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec63eb2-c4da-4174-9ed8-9bb86d19b60d" containerName="cleanup"
Nov 03 12:30:32.773561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.773542    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.812596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.812562    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.812775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.812628    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48gc\" (UniqueName: \"kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.812775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.812709    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.822862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.822840    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:30:32.869025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.868997    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:30:32.869025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.869003    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:30:32.869198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.869007    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:30:32.872616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.872598    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"]
Nov 03 12:30:32.914189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.914154    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.914374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.914212    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48gc\" (UniqueName: \"kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.914374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.914284    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.915078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.915057    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.916699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.916680    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:32.967275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:32.967244    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48gc\" (UniqueName: \"kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc\") pod \"maas-api-revoke-sub-45378f68-6fd54\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:33.087625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:33.087540    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:33.456590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:33.456559    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"]
Nov 03 12:30:33.457560 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:33.457532    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4557a45_fdf5_4e0c_bf62_2ca5cd4d5295.slice/crio-617a32e4e77ccc5bb7f085b0a7043c3c1e80913ebad14f472d96be67592092dd WatchSource:0}: Error finding container 617a32e4e77ccc5bb7f085b0a7043c3c1e80913ebad14f472d96be67592092dd: Status 404 returned error can't find the container with id 617a32e4e77ccc5bb7f085b0a7043c3c1e80913ebad14f472d96be67592092dd
Nov 03 12:30:34.111952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:34.111855    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54" event={"ID":"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295","Type":"ContainerStarted","Data":"c4d57b6a6219c4dc835043a2f4668dfd81db782e09eb8e4a45cd4b386b6bd1b6"}
Nov 03 12:30:34.111952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:34.111912    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54" event={"ID":"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295","Type":"ContainerStarted","Data":"617a32e4e77ccc5bb7f085b0a7043c3c1e80913ebad14f472d96be67592092dd"}
Nov 03 12:30:34.221132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:34.221073    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54" podStartSLOduration=2.221059429 podStartE2EDuration="2.221059429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:34.217186413 +0000 UTC m=+2375.788770354" watchObservedRunningTime="2026-11-03 12:30:34.221059429 +0000 UTC m=+2375.792643340"
Nov 03 12:30:35.119779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:35.119730    4546 generic.go:356] "Generic (PLEG): container finished" podID="a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" containerID="c4d57b6a6219c4dc835043a2f4668dfd81db782e09eb8e4a45cd4b386b6bd1b6" exitCode=0
Nov 03 12:30:35.120188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:35.119779    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54" event={"ID":"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295","Type":"ContainerDied","Data":"c4d57b6a6219c4dc835043a2f4668dfd81db782e09eb8e4a45cd4b386b6bd1b6"}
Nov 03 12:30:36.279105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.279080    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:36.356663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.356627    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca\") pod \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") "
Nov 03 12:30:36.356881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.356685    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc\" (UniqueName: \"kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc\") pod \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") "
Nov 03 12:30:36.357025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.357007    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca\") pod \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\" (UID: \"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295\") "
Nov 03 12:30:36.357134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.357103    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca" pod "a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" (UID: "a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:36.357388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.357371    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:36.359210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.359188    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca" pod "a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" (UID: "a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:36.359410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.359386    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc" pod "a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" (UID: "a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295"). InnerVolumeSpecName "kube-api-access-r48gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:36.458698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.458619    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:36.458698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:36.458646    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r48gc\" (UniqueName: \"kubernetes.io/projected/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295-kube-api-access-r48gc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:37.134840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:37.134812    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"
Nov 03 12:30:37.135030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:37.134825    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54" event={"ID":"a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295","Type":"ContainerDied","Data":"617a32e4e77ccc5bb7f085b0a7043c3c1e80913ebad14f472d96be67592092dd"}
Nov 03 12:30:37.135030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:37.134883    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="617a32e4e77ccc5bb7f085b0a7043c3c1e80913ebad14f472d96be67592092dd"
Nov 03 12:30:37.651837 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:30:37.651732    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:30:41.656191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.656158    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"]
Nov 03 12:30:41.656904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.656884    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" containerName="revoke-keys"
Nov 03 12:30:41.657004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.656908    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" containerName="revoke-keys"
Nov 03 12:30:41.657107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.657090    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" containerName="revoke-keys"
Nov 03 12:30:41.660731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.660713    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.719583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.719551    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.719788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.719616    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.719788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.719740    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.719927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.719876    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.719983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.719928    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.720035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.719998    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.720091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.720040    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvxt\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.720139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.720123    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.720194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.720151    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.759610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.759578    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"]
Nov 03 12:30:41.821654 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821626    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.821866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821676    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.821866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821700    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.821866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821738    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.821866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821800    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvxt\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.821866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821856    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821887    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.821940    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.822097    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.822163    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.822292    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.822297    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.822441    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.822546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.822525    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.824632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.824613    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.824956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.824932    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.892536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.892499    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.892713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.892591    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvxt\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:41.974203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:41.974145    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:42.430496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:42.430461    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"]
Nov 03 12:30:42.433871 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:42.433828    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf06f6d5_420f_4f6a_9103_ccff5cd0b392.slice/crio-0e9dd90392715332dbf9fe794043a822707b1dff70d98002efc431d7c8043336 WatchSource:0}: Error finding container 0e9dd90392715332dbf9fe794043a822707b1dff70d98002efc431d7c8043336: Status 404 returned error can't find the container with id 0e9dd90392715332dbf9fe794043a822707b1dff70d98002efc431d7c8043336
Nov 03 12:30:42.436449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:42.436412    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:30:42.436531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:42.436498    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:30:42.436569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:42.436541    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:30:43.182168 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:43.182126    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" event={"ID":"cf06f6d5-420f-4f6a-9103-ccff5cd0b392","Type":"ContainerStarted","Data":"a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081"}
Nov 03 12:30:43.182691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:43.182196    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" event={"ID":"cf06f6d5-420f-4f6a-9103-ccff5cd0b392","Type":"ContainerStarted","Data":"0e9dd90392715332dbf9fe794043a822707b1dff70d98002efc431d7c8043336"}
Nov 03 12:30:43.292323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:43.292267    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podStartSLOduration=2.292247296 podStartE2EDuration="2.292247296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:43.292041386 +0000 UTC m=+2384.863625312" watchObservedRunningTime="2026-11-03 12:30:43.292247296 +0000 UTC m=+2384.863831209"
Nov 03 12:30:43.975080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:43.975042    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:43.976873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:43.976849    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 12:30:43.977016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:43.976897    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 12:30:44.974517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:44.974483    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 12:30:44.975039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:44.974536    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 12:30:45.974933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:45.974898    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 12:30:45.975380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:45.974980    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 12:30:46.975400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:46.975363    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 12:30:46.975956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:46.975430    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 12:30:47.974918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:47.974880    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 12:30:47.975106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:47.974941    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 12:30:48.975065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:48.975032    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 12:30:48.975464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:48.975085    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 12:30:49.979568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:49.979538    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:49.980032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:49.980018    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:49.980827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:49.980806    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:30:51.028116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.028079    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"]
Nov 03 12:30:51.029008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.028969    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" podUID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" containerName="payload-processing" containerID="cri-o://5973ac90ced3bfb43ac07a84b6924c9676e3fbb10b78a9ea472e7e6bd289ccbd" gracePeriod=30
Nov 03 12:30:51.080346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.080314    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"]
Nov 03 12:30:51.080555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.080526    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz" podUID="6d6bc69a-bfa7-4158-8318-d7573a40b1b0" containerName="payload-pre-processing" containerID="cri-o://017347fd6ff7f4aeef4d374c10fec573ecfad99d6863b571bc6f3c89241dfd47" gracePeriod=30
Nov 03 12:30:51.282593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.282521    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d6bc69a-bfa7-4158-8318-d7573a40b1b0" containerID="017347fd6ff7f4aeef4d374c10fec573ecfad99d6863b571bc6f3c89241dfd47" exitCode=0
Nov 03 12:30:51.282729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.282618    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz" event={"ID":"6d6bc69a-bfa7-4158-8318-d7573a40b1b0","Type":"ContainerDied","Data":"017347fd6ff7f4aeef4d374c10fec573ecfad99d6863b571bc6f3c89241dfd47"}
Nov 03 12:30:51.284502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.284478    4546 generic.go:356] "Generic (PLEG): container finished" podID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" containerID="5973ac90ced3bfb43ac07a84b6924c9676e3fbb10b78a9ea472e7e6bd289ccbd" exitCode=0
Nov 03 12:30:51.284848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.284826    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" event={"ID":"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef","Type":"ContainerDied","Data":"5973ac90ced3bfb43ac07a84b6924c9676e3fbb10b78a9ea472e7e6bd289ccbd"}
Nov 03 12:30:51.369588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.369565    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:30:51.405728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.405705    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:30:51.486191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.486163    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume\") pod \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") "
Nov 03 12:30:51.486373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.486306    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz\" (UniqueName: \"kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz\") pod \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\" (UID: \"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef\") "
Nov 03 12:30:51.486452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.486367    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw\" (UniqueName: \"kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw\") pod \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") "
Nov 03 12:30:51.486452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.486419    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume\") pod \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\" (UID: \"6d6bc69a-bfa7-4158-8318-d7573a40b1b0\") "
Nov 03 12:30:51.486669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.486532    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume" pod "f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" (UID: "f0592ed8-db55-4ca6-b7fa-7fd0608d59ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:51.487103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.486972    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume" pod "6d6bc69a-bfa7-4158-8318-d7573a40b1b0" (UID: "6d6bc69a-bfa7-4158-8318-d7573a40b1b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:51.487209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.487041    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.488808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.488780    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz" pod "f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" (UID: "f0592ed8-db55-4ca6-b7fa-7fd0608d59ef"). InnerVolumeSpecName "kube-api-access-d75fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:51.489026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.488992    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw" pod "6d6bc69a-bfa7-4158-8318-d7573a40b1b0" (UID: "6d6bc69a-bfa7-4158-8318-d7573a40b1b0"). InnerVolumeSpecName "kube-api-access-m7mjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:51.589130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.589045    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75fz\" (UniqueName: \"kubernetes.io/projected/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef-kube-api-access-d75fz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.589130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.589081    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7mjw\" (UniqueName: \"kubernetes.io/projected/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-kube-api-access-m7mjw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:51.589130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:51.589117    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6bc69a-bfa7-4158-8318-d7573a40b1b0-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:52.293400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.293369    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"
Nov 03 12:30:52.293888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.293376    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz" event={"ID":"6d6bc69a-bfa7-4158-8318-d7573a40b1b0","Type":"ContainerDied","Data":"f32742b14063e6ec495abc2453eb60b73bb2a0e81afdff5e0ebf20ce3af8a99c"}
Nov 03 12:30:52.293888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.293517    4546 scope.go:122] "RemoveContainer" containerID="017347fd6ff7f4aeef4d374c10fec573ecfad99d6863b571bc6f3c89241dfd47"
Nov 03 12:30:52.295009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.294914    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"
Nov 03 12:30:52.295009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.294955    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" event={"ID":"f0592ed8-db55-4ca6-b7fa-7fd0608d59ef","Type":"ContainerDied","Data":"b47ceca6d8a8836889d0114f709e0306efc186a49e80a4001d2613f8767d05a2"}
Nov 03 12:30:52.305688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.305667    4546 scope.go:122] "RemoveContainer" containerID="5973ac90ced3bfb43ac07a84b6924c9676e3fbb10b78a9ea472e7e6bd289ccbd"
Nov 03 12:30:52.308922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.308880    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6" podUID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.174:9004: i/o timeout"
Nov 03 12:30:52.445663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.445630    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"]
Nov 03 12:30:52.449128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.449100    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f3eb2-5c878c8f5-jgrvz"]
Nov 03 12:30:52.550103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.550058    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"]
Nov 03 12:30:52.553400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.553376    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f3eb2-68649599c8-gfmj6"]
Nov 03 12:30:52.573893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.573829    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6bc69a-bfa7-4158-8318-d7573a40b1b0" path="/var/lib/kubelet/pods/6d6bc69a-bfa7-4158-8318-d7573a40b1b0/volumes"
Nov 03 12:30:52.574298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:52.574284    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" path="/var/lib/kubelet/pods/f0592ed8-db55-4ca6-b7fa-7fd0608d59ef/volumes"
Nov 03 12:30:53.850377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.850333    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"]
Nov 03 12:30:53.851151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.851129    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" containerName="payload-processing"
Nov 03 12:30:53.851260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.851153    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" containerName="payload-processing"
Nov 03 12:30:53.851260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.851178    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d6bc69a-bfa7-4158-8318-d7573a40b1b0" containerName="payload-pre-processing"
Nov 03 12:30:53.851260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.851186    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6bc69a-bfa7-4158-8318-d7573a40b1b0" containerName="payload-pre-processing"
Nov 03 12:30:53.851432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.851341    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0592ed8-db55-4ca6-b7fa-7fd0608d59ef" containerName="payload-processing"
Nov 03 12:30:53.851432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.851356    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6bc69a-bfa7-4158-8318-d7573a40b1b0" containerName="payload-pre-processing"
Nov 03 12:30:53.856256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.856234    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:53.904908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.904876    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:30:53.908864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.908830    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"]
Nov 03 12:30:53.920651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.920618    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:53.920879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.920704    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:53.920879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.920841    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb8t\" (UniqueName: \"kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:53.951025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.950987    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:30:53.951025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.950989    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:30:53.951214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:53.951140    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:30:54.022315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.022283    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb8t\" (UniqueName: \"kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.022520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.022405    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.022520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.022438    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.023081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.023049    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.024948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.024929    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.074850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.074816    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb8t\" (UniqueName: \"kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t\") pod \"maas-api-revoke-sub-217d3cbd-8xd2f\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.169373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.169297    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:54.540876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:54.540846    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"]
Nov 03 12:30:54.541459 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:30:54.541435    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0facd28e_cf85_433d_a4f6_c102cc8862ca.slice/crio-b202819baa5c6d8f8eb08467dc0ef619ab46d7f43353912ab2da500babfa6f41 WatchSource:0}: Error finding container b202819baa5c6d8f8eb08467dc0ef619ab46d7f43353912ab2da500babfa6f41: Status 404 returned error can't find the container with id b202819baa5c6d8f8eb08467dc0ef619ab46d7f43353912ab2da500babfa6f41
Nov 03 12:30:55.318554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:55.318515    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f" event={"ID":"0facd28e-cf85-433d-a4f6-c102cc8862ca","Type":"ContainerStarted","Data":"23539fc5760ad19a4f777a8704050895896730a68551067e2fafc498b59d8c6e"}
Nov 03 12:30:55.319169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:55.318576    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f" event={"ID":"0facd28e-cf85-433d-a4f6-c102cc8862ca","Type":"ContainerStarted","Data":"b202819baa5c6d8f8eb08467dc0ef619ab46d7f43353912ab2da500babfa6f41"}
Nov 03 12:30:55.423211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:55.423157    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f" podStartSLOduration=2.423141533 podStartE2EDuration="2.423141533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:55.422735294 +0000 UTC m=+2396.994319206" watchObservedRunningTime="2026-11-03 12:30:55.423141533 +0000 UTC m=+2396.994725444"
Nov 03 12:30:56.327419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:56.327385    4546 generic.go:356] "Generic (PLEG): container finished" podID="0facd28e-cf85-433d-a4f6-c102cc8862ca" containerID="23539fc5760ad19a4f777a8704050895896730a68551067e2fafc498b59d8c6e" exitCode=0
Nov 03 12:30:56.327858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:56.327469    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f" event={"ID":"0facd28e-cf85-433d-a4f6-c102cc8862ca","Type":"ContainerDied","Data":"23539fc5760ad19a4f777a8704050895896730a68551067e2fafc498b59d8c6e"}
Nov 03 12:30:57.501856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.501826    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:30:57.668828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.668210    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca\") pod \"0facd28e-cf85-433d-a4f6-c102cc8862ca\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") "
Nov 03 12:30:57.668828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.668265    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t\" (UniqueName: \"kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t\") pod \"0facd28e-cf85-433d-a4f6-c102cc8862ca\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") "
Nov 03 12:30:57.668828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.668509    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca\") pod \"0facd28e-cf85-433d-a4f6-c102cc8862ca\" (UID: \"0facd28e-cf85-433d-a4f6-c102cc8862ca\") "
Nov 03 12:30:57.691502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.671451    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca" pod "0facd28e-cf85-433d-a4f6-c102cc8862ca" (UID: "0facd28e-cf85-433d-a4f6-c102cc8862ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:57.691502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.682212    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t" pod "0facd28e-cf85-433d-a4f6-c102cc8862ca" (UID: "0facd28e-cf85-433d-a4f6-c102cc8862ca"). InnerVolumeSpecName "kube-api-access-lxb8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:57.694957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.694901    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca" pod "0facd28e-cf85-433d-a4f6-c102cc8862ca" (UID: "0facd28e-cf85-433d-a4f6-c102cc8862ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:57.769893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.769859    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:57.769893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.769887    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0facd28e-cf85-433d-a4f6-c102cc8862ca-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:57.769893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:57.769897    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxb8t\" (UniqueName: \"kubernetes.io/projected/0facd28e-cf85-433d-a4f6-c102cc8862ca-kube-api-access-lxb8t\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:58.342223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:58.342181    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f" event={"ID":"0facd28e-cf85-433d-a4f6-c102cc8862ca","Type":"ContainerDied","Data":"b202819baa5c6d8f8eb08467dc0ef619ab46d7f43353912ab2da500babfa6f41"}
Nov 03 12:30:58.342417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:58.342285    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b202819baa5c6d8f8eb08467dc0ef619ab46d7f43353912ab2da500babfa6f41"
Nov 03 12:30:58.342417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:30:58.342195    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"
Nov 03 12:31:00.409790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:00.409734    4546 scope.go:122] "RemoveContainer" containerID="366532d9b50871ac5307f803f66442e1ddeed14a6427a795cc7540df74cba87f"
Nov 03 12:31:00.430571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:00.430548    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"]
Nov 03 12:31:00.430877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:00.430837    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" containerID="cri-o://2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca" gracePeriod=30
Nov 03 12:31:03.185328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:03.185292    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:31:03.185738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:03.185347    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:31:03.493966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:03.493935    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"]
Nov 03 12:31:03.494170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:03.494150    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy" containerID="cri-o://2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec" gracePeriod=30
Nov 03 12:31:05.705046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.704986    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:31:05.807988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.807955    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808009    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808051    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808090    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808145    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808175    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808238    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808266    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808325    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808340    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808393    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert\") pod \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\" (UID: \"b2f3b83c-ac0b-47c3-93a5-441f9d349054\") "
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808432    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:05.808466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808449    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:05.808872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808846    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:05.808963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808945    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.809035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808978    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.809035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.808993    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.809035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.809008    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.811244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.811214    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:31:05.811377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.811254    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:05.811377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.811339    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "kube-api-access-gdz9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:05.811377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.811371    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token" pod "b2f3b83c-ac0b-47c3-93a5-441f9d349054" (UID: "b2f3b83c-ac0b-47c3-93a5-441f9d349054"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:05.910948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.910919    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.910948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.910948    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdz9b\" (UniqueName: \"kubernetes.io/projected/b2f3b83c-ac0b-47c3-93a5-441f9d349054-kube-api-access-gdz9b\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.911143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.910959    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.911143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.910967    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:05.911143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:05.910976    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f3b83c-ac0b-47c3-93a5-441f9d349054-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:06.394870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.394835    4546 generic.go:356] "Generic (PLEG): container finished" podID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerID="2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca" exitCode=0
Nov 03 12:31:06.395070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.394895    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"
Nov 03 12:31:06.395070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.394918    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" event={"ID":"b2f3b83c-ac0b-47c3-93a5-441f9d349054","Type":"ContainerDied","Data":"2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca"}
Nov 03 12:31:06.395070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.394995    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9" event={"ID":"b2f3b83c-ac0b-47c3-93a5-441f9d349054","Type":"ContainerDied","Data":"3bee9afc82de8a54a36984e786d971b99f8eb4ac2fd6f081c35bc41b0311942d"}
Nov 03 12:31:06.395070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.395020    4546 scope.go:122] "RemoveContainer" containerID="2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca"
Nov 03 12:31:06.405395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.405378    4546 scope.go:122] "RemoveContainer" containerID="7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"
Nov 03 12:31:06.415923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.415715    4546 scope.go:122] "RemoveContainer" containerID="2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca"
Nov 03 12:31:06.416953 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:06.416777    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca\": container with ID starting with 2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca not found: ID does not exist" containerID="2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca"
Nov 03 12:31:06.416953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.416816    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca"} err="failed to get container status \"2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca\": rpc error: code = NotFound desc = could not find container \"2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca\": container with ID starting with 2bed005d11c71b65c5dae52cc78e87c52aead5c66624b339f02c7a7b26f2b0ca not found: ID does not exist"
Nov 03 12:31:06.416953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.416858    4546 scope.go:122] "RemoveContainer" containerID="7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"
Nov 03 12:31:06.417729 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:06.417697    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10\": container with ID starting with 7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10 not found: ID does not exist" containerID="7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"
Nov 03 12:31:06.417854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.417740    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10"} err="failed to get container status \"7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10\": rpc error: code = NotFound desc = could not find container \"7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10\": container with ID starting with 7067eee029b32f448109004cbf2f077752826691c9140ba267d779c7cfb8be10 not found: ID does not exist"
Nov 03 12:31:06.594059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.594026    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"]
Nov 03 12:31:06.639916 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:06.639879    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f3eb2-openshift-default-6f9bb94bb7-pvvm9"]
Nov 03 12:31:07.654110 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:07.654078    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:31:08.573165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.573137    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" path="/var/lib/kubelet/pods/b2f3b83c-ac0b-47c3-93a5-441f9d349054/volumes"
Nov 03 12:31:08.779535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.779512    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:31:08.847817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.847780    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.847857    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.847928    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.847955    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.847994    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848024    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848054    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848084    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848160    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:08.848238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848208    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket\") pod \"8f37d690-8f54-4ec0-a146-0b8e7385c277\" (UID: \"8f37d690-8f54-4ec0-a146-0b8e7385c277\") "
Nov 03 12:31:08.848516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848400    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:08.848516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848425    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:08.848516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848423    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:08.848673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848586    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:08.848817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848799    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.848817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848818    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.848992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848827    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37d690-8f54-4ec0-a146-0b8e7385c277-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.848992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848835    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.848992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.848865    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.851081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.851055    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:31:08.851190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.851080    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:08.851190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.851106    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "kube-api-access-cvl96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:08.851312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.851216    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy" pod "8f37d690-8f54-4ec0-a146-0b8e7385c277" (UID: "8f37d690-8f54-4ec0-a146-0b8e7385c277"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:08.950636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.950601    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvl96\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-kube-api-access-cvl96\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.950636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.950633    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.950636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.950645    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:08.950898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:08.950653    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37d690-8f54-4ec0-a146-0b8e7385c277-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:09.418645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.418611    4546 generic.go:356] "Generic (PLEG): container finished" podID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerID="2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec" exitCode=0
Nov 03 12:31:09.418827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.418663    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"
Nov 03 12:31:09.418827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.418705    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" event={"ID":"8f37d690-8f54-4ec0-a146-0b8e7385c277","Type":"ContainerDied","Data":"2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec"}
Nov 03 12:31:09.418827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.418775    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr" event={"ID":"8f37d690-8f54-4ec0-a146-0b8e7385c277","Type":"ContainerDied","Data":"19e688b6a1418fe51325f1a8e6d1dd2234b8d82d2123e7621bbac319b8e6c224"}
Nov 03 12:31:09.418827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.418802    4546 scope.go:122] "RemoveContainer" containerID="2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec"
Nov 03 12:31:09.429495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.429474    4546 scope.go:122] "RemoveContainer" containerID="ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"
Nov 03 12:31:09.438431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.438414    4546 scope.go:122] "RemoveContainer" containerID="2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec"
Nov 03 12:31:09.438680 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:09.438658    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec\": container with ID starting with 2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec not found: ID does not exist" containerID="2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec"
Nov 03 12:31:09.438804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.438685    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec"} err="failed to get container status \"2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec\": rpc error: code = NotFound desc = could not find container \"2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec\": container with ID starting with 2d6cca59b0de2c200387f693dc54fd3eb1a3874a40a8b563bb556db3714e92ec not found: ID does not exist"
Nov 03 12:31:09.438804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.438706    4546 scope.go:122] "RemoveContainer" containerID="ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"
Nov 03 12:31:09.438984 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:09.438963    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df\": container with ID starting with ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df not found: ID does not exist" containerID="ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"
Nov 03 12:31:09.439031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.438991    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df"} err="failed to get container status \"ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df\": rpc error: code = NotFound desc = could not find container \"ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df\": container with ID starting with ee12ff6f2acf422a8f92255af3070d6e064c3179ab5ee074122071165c8937df not found: ID does not exist"
Nov 03 12:31:09.573265 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.573234    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"]
Nov 03 12:31:09.620879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:09.620848    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-04d73d-openshift-default-5c7f8fcfb5-rwrqr"]
Nov 03 12:31:10.573208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:10.573172    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" path="/var/lib/kubelet/pods/8f37d690-8f54-4ec0-a146-0b8e7385c277/volumes"
Nov 03 12:31:11.897465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.897422    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"]
Nov 03 12:31:11.897982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.897968    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy"
Nov 03 12:31:11.898048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.897983    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy"
Nov 03 12:31:11.898048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.897998    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0facd28e-cf85-433d-a4f6-c102cc8862ca" containerName="revoke-keys"
Nov 03 12:31:11.898048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898002    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0facd28e-cf85-433d-a4f6-c102cc8862ca" containerName="revoke-keys"
Nov 03 12:31:11.898048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898028    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy"
Nov 03 12:31:11.898048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898033    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy"
Nov 03 12:31:11.898048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898050    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy"
Nov 03 12:31:11.898238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898055    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy"
Nov 03 12:31:11.898238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898143    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy"
Nov 03 12:31:11.898238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898154    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy"
Nov 03 12:31:11.898238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898161    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0facd28e-cf85-433d-a4f6-c102cc8862ca" containerName="revoke-keys"
Nov 03 12:31:11.898238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898171    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy"
Nov 03 12:31:11.898238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.898179    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f3b83c-ac0b-47c3-93a5-441f9d349054" containerName="istio-proxy"
Nov 03 12:31:11.902998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.902980    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:11.955943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.955909    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"]
Nov 03 12:31:11.987814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.987779    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qfw\" (UniqueName: \"kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:11.987975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.987950    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:11.988029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.988013    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:11.998030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.997965    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:31:11.998030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.997965    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:31:11.998030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.998001    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:31:11.998268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:11.998060    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:31:12.088695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.088661    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qfw\" (UniqueName: \"kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.088874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.088785    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.088874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.088839    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.089425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.089402    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.091338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.091317    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.141267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.141231    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qfw\" (UniqueName: \"kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw\") pod \"maas-api-revoke-sub-58866b73-2rjtm\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.216466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.216433    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:12.600735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.600687    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"]
Nov 03 12:31:12.606040 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:12.606005    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162c39ae_9d38_4fb7_8599_dd6d897db1f7.slice/crio-cec1c30fcb947da3ab04d14f64cbbdbab79d0fe16c2ffb4e26f0098394ce2e05 WatchSource:0}: Error finding container cec1c30fcb947da3ab04d14f64cbbdbab79d0fe16c2ffb4e26f0098394ce2e05: Status 404 returned error can't find the container with id cec1c30fcb947da3ab04d14f64cbbdbab79d0fe16c2ffb4e26f0098394ce2e05
Nov 03 12:31:12.889653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.889575    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"]
Nov 03 12:31:12.890173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.890158    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy"
Nov 03 12:31:12.890231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.890174    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f37d690-8f54-4ec0-a146-0b8e7385c277" containerName="istio-proxy"
Nov 03 12:31:12.895053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.895036    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:12.947096 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:12.947062    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"]
Nov 03 12:31:13.009852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.009818    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.010021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.009979    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.010021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.010019    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6w6\" (UniqueName: \"kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.111440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.111406    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.111600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.111507    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.111600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.111539    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6w6\" (UniqueName: \"kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.112085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.112053    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.114021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.113998    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.164498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.164421    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6w6\" (UniqueName: \"kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6\") pod \"maas-api-revoke-sub-955e1cab-2xkcp\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.208606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.208574    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:13.446893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.446801    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm" event={"ID":"162c39ae-9d38-4fb7-8599-dd6d897db1f7","Type":"ContainerStarted","Data":"60826e700f00a0097f7fcb56586e0379b37d4c1e24a9c8986e00f1bf56c28ec2"}
Nov 03 12:31:13.446893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.446862    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm" event={"ID":"162c39ae-9d38-4fb7-8599-dd6d897db1f7","Type":"ContainerStarted","Data":"cec1c30fcb947da3ab04d14f64cbbdbab79d0fe16c2ffb4e26f0098394ce2e05"}
Nov 03 12:31:13.574154 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:13.574122    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5515370d_003a_4f7e_98b1_beb01bf18842.slice/crio-23b8f69f4661593132e6469746140b5a03c74233c24640cbcb89c819d1f937c6 WatchSource:0}: Error finding container 23b8f69f4661593132e6469746140b5a03c74233c24640cbcb89c819d1f937c6: Status 404 returned error can't find the container with id 23b8f69f4661593132e6469746140b5a03c74233c24640cbcb89c819d1f937c6
Nov 03 12:31:13.598945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.598900    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm" podStartSLOduration=2.59888604 podStartE2EDuration="2.59888604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:13.550490125 +0000 UTC m=+2415.122074060" watchObservedRunningTime="2026-11-03 12:31:13.59888604 +0000 UTC m=+2415.170469951"
Nov 03 12:31:13.603442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:13.603418    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"]
Nov 03 12:31:14.455786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:14.455672    4546 generic.go:356] "Generic (PLEG): container finished" podID="162c39ae-9d38-4fb7-8599-dd6d897db1f7" containerID="60826e700f00a0097f7fcb56586e0379b37d4c1e24a9c8986e00f1bf56c28ec2" exitCode=0
Nov 03 12:31:14.455786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:14.455740    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm" event={"ID":"162c39ae-9d38-4fb7-8599-dd6d897db1f7","Type":"ContainerDied","Data":"60826e700f00a0097f7fcb56586e0379b37d4c1e24a9c8986e00f1bf56c28ec2"}
Nov 03 12:31:14.457379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:14.457348    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp" event={"ID":"5515370d-003a-4f7e-98b1-beb01bf18842","Type":"ContainerStarted","Data":"c479b3447a26b01e512640a2a903f2c8f32f0a1943e0317db6024242ee19ce0a"}
Nov 03 12:31:14.457517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:14.457399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp" event={"ID":"5515370d-003a-4f7e-98b1-beb01bf18842","Type":"ContainerStarted","Data":"23b8f69f4661593132e6469746140b5a03c74233c24640cbcb89c819d1f937c6"}
Nov 03 12:31:14.668234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:14.668169    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp" podStartSLOduration=2.6681545939999998 podStartE2EDuration="2.668154594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:14.665632383 +0000 UTC m=+2416.237216295" watchObservedRunningTime="2026-11-03 12:31:14.668154594 +0000 UTC m=+2416.239738506"
Nov 03 12:31:15.466567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.466533    4546 generic.go:356] "Generic (PLEG): container finished" podID="5515370d-003a-4f7e-98b1-beb01bf18842" containerID="c479b3447a26b01e512640a2a903f2c8f32f0a1943e0317db6024242ee19ce0a" exitCode=0
Nov 03 12:31:15.467088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.466634    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp" event={"ID":"5515370d-003a-4f7e-98b1-beb01bf18842","Type":"ContainerDied","Data":"c479b3447a26b01e512640a2a903f2c8f32f0a1943e0317db6024242ee19ce0a"}
Nov 03 12:31:15.622429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.622388    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"]
Nov 03 12:31:15.626677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.626651    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.640117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.640090    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:15.681710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.681674    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"]
Nov 03 12:31:15.746029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.745933    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca\") pod \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") "
Nov 03 12:31:15.746029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.745976    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw\" (UniqueName: \"kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw\") pod \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") "
Nov 03 12:31:15.746282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.746185    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca\") pod \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\" (UID: \"162c39ae-9d38-4fb7-8599-dd6d897db1f7\") "
Nov 03 12:31:15.746525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.746491    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.746525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.746496    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca" pod "162c39ae-9d38-4fb7-8599-dd6d897db1f7" (UID: "162c39ae-9d38-4fb7-8599-dd6d897db1f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:15.746697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.746676    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcjt\" (UniqueName: \"kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.746776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.746734    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.746990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.746974    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:15.748900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.748880    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca" pod "162c39ae-9d38-4fb7-8599-dd6d897db1f7" (UID: "162c39ae-9d38-4fb7-8599-dd6d897db1f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:15.748900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.748888    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw" pod "162c39ae-9d38-4fb7-8599-dd6d897db1f7" (UID: "162c39ae-9d38-4fb7-8599-dd6d897db1f7"). InnerVolumeSpecName "kube-api-access-45qfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:15.847792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.847734    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.847967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.847868    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcjt\" (UniqueName: \"kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.847967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.847907    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.848108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.848002    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162c39ae-9d38-4fb7-8599-dd6d897db1f7-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:15.848108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.848017    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qfw\" (UniqueName: \"kubernetes.io/projected/162c39ae-9d38-4fb7-8599-dd6d897db1f7-kube-api-access-45qfw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:15.848462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.848441    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.850817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.850788    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.901054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.901022    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcjt\" (UniqueName: \"kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt\") pod \"maas-api-revoke-sub-b8ceb6b1-vp9qb\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:15.950007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:15.949970    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:16.321271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.321241    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"]
Nov 03 12:31:16.322580 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:16.322549    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c81586_3b88_418d_a3f5_742caf20efe4.slice/crio-39601fc02537b74789bee1e3a6a1d87d96a799d71156873cdf350c62d6dca431 WatchSource:0}: Error finding container 39601fc02537b74789bee1e3a6a1d87d96a799d71156873cdf350c62d6dca431: Status 404 returned error can't find the container with id 39601fc02537b74789bee1e3a6a1d87d96a799d71156873cdf350c62d6dca431
Nov 03 12:31:16.475251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.475222    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"
Nov 03 12:31:16.475733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.475255    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm" event={"ID":"162c39ae-9d38-4fb7-8599-dd6d897db1f7","Type":"ContainerDied","Data":"cec1c30fcb947da3ab04d14f64cbbdbab79d0fe16c2ffb4e26f0098394ce2e05"}
Nov 03 12:31:16.475733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.475305    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec1c30fcb947da3ab04d14f64cbbdbab79d0fe16c2ffb4e26f0098394ce2e05"
Nov 03 12:31:16.476814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.476790    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb" event={"ID":"84c81586-3b88-418d-a3f5-742caf20efe4","Type":"ContainerStarted","Data":"39601fc02537b74789bee1e3a6a1d87d96a799d71156873cdf350c62d6dca431"}
Nov 03 12:31:16.630649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.630622    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:16.760438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.760408    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca\") pod \"5515370d-003a-4f7e-98b1-beb01bf18842\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") "
Nov 03 12:31:16.760624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.760590    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca\") pod \"5515370d-003a-4f7e-98b1-beb01bf18842\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") "
Nov 03 12:31:16.760682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.760628    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6\" (UniqueName: \"kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6\") pod \"5515370d-003a-4f7e-98b1-beb01bf18842\" (UID: \"5515370d-003a-4f7e-98b1-beb01bf18842\") "
Nov 03 12:31:16.760804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.760775    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca" pod "5515370d-003a-4f7e-98b1-beb01bf18842" (UID: "5515370d-003a-4f7e-98b1-beb01bf18842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:16.761011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.760994    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:16.762986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.762962    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6" pod "5515370d-003a-4f7e-98b1-beb01bf18842" (UID: "5515370d-003a-4f7e-98b1-beb01bf18842"). InnerVolumeSpecName "kube-api-access-gr6w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:16.763089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.763049    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca" pod "5515370d-003a-4f7e-98b1-beb01bf18842" (UID: "5515370d-003a-4f7e-98b1-beb01bf18842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:16.862528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.862438    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5515370d-003a-4f7e-98b1-beb01bf18842-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:16.862528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:16.862471    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr6w6\" (UniqueName: \"kubernetes.io/projected/5515370d-003a-4f7e-98b1-beb01bf18842-kube-api-access-gr6w6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:17.486525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:17.486492    4546 generic.go:356] "Generic (PLEG): container finished" podID="84c81586-3b88-418d-a3f5-742caf20efe4" containerID="25d8a7d36a575ae26c3ec4b01054cbe45cafaad1df6a11000259b5fbad6c35f9" exitCode=0
Nov 03 12:31:17.487041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:17.486570    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb" event={"ID":"84c81586-3b88-418d-a3f5-742caf20efe4","Type":"ContainerDied","Data":"25d8a7d36a575ae26c3ec4b01054cbe45cafaad1df6a11000259b5fbad6c35f9"}
Nov 03 12:31:17.488120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:17.488100    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"
Nov 03 12:31:17.488255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:17.488132    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp" event={"ID":"5515370d-003a-4f7e-98b1-beb01bf18842","Type":"ContainerDied","Data":"23b8f69f4661593132e6469746140b5a03c74233c24640cbcb89c819d1f937c6"}
Nov 03 12:31:17.488255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:17.488178    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b8f69f4661593132e6469746140b5a03c74233c24640cbcb89c819d1f937c6"
Nov 03 12:31:18.654891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.654861    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:18.788164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.788085    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca\") pod \"84c81586-3b88-418d-a3f5-742caf20efe4\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") "
Nov 03 12:31:18.788329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.788199    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt\" (UniqueName: \"kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt\") pod \"84c81586-3b88-418d-a3f5-742caf20efe4\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") "
Nov 03 12:31:18.788329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.788276    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca\") pod \"84c81586-3b88-418d-a3f5-742caf20efe4\" (UID: \"84c81586-3b88-418d-a3f5-742caf20efe4\") "
Nov 03 12:31:18.788671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.788641    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca" pod "84c81586-3b88-418d-a3f5-742caf20efe4" (UID: "84c81586-3b88-418d-a3f5-742caf20efe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:18.790568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.790539    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca" pod "84c81586-3b88-418d-a3f5-742caf20efe4" (UID: "84c81586-3b88-418d-a3f5-742caf20efe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:18.790934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.790907    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt" pod "84c81586-3b88-418d-a3f5-742caf20efe4" (UID: "84c81586-3b88-418d-a3f5-742caf20efe4"). InnerVolumeSpecName "kube-api-access-bpcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:18.810247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810219    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"]
Nov 03 12:31:18.810844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810830    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5515370d-003a-4f7e-98b1-beb01bf18842" containerName="revoke-keys"
Nov 03 12:31:18.810901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810846    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5515370d-003a-4f7e-98b1-beb01bf18842" containerName="revoke-keys"
Nov 03 12:31:18.810901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810872    4546 cpu_manager.go:414] "containerMap: removing container" podUID="162c39ae-9d38-4fb7-8599-dd6d897db1f7" containerName="revoke-keys"
Nov 03 12:31:18.810901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810877    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162c39ae-9d38-4fb7-8599-dd6d897db1f7" containerName="revoke-keys"
Nov 03 12:31:18.810901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810884    4546 cpu_manager.go:414] "containerMap: removing container" podUID="84c81586-3b88-418d-a3f5-742caf20efe4" containerName="revoke-keys"
Nov 03 12:31:18.810901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810889    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c81586-3b88-418d-a3f5-742caf20efe4" containerName="revoke-keys"
Nov 03 12:31:18.811060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810976    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5515370d-003a-4f7e-98b1-beb01bf18842" containerName="revoke-keys"
Nov 03 12:31:18.811060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810991    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="162c39ae-9d38-4fb7-8599-dd6d897db1f7" containerName="revoke-keys"
Nov 03 12:31:18.811060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.810997    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c81586-3b88-418d-a3f5-742caf20efe4" containerName="revoke-keys"
Nov 03 12:31:18.814679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.814658    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.869882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.869847    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"]
Nov 03 12:31:18.890195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.890161    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.890372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.890211    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.890372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.890310    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgwg\" (UniqueName: \"kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.890455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.890405    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:18.890455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.890422    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c81586-3b88-418d-a3f5-742caf20efe4-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:18.890455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.890432    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcjt\" (UniqueName: \"kubernetes.io/projected/84c81586-3b88-418d-a3f5-742caf20efe4-kube-api-access-bpcjt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:18.991688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.991644    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.991908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.991700    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.991908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.991795    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgwg\" (UniqueName: \"kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.992363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.992330    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:18.994127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:18.994107    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:19.044665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.044592    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgwg\" (UniqueName: \"kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg\") pod \"maas-api-revoke-sub-8a1b9bcb-92r7q\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:19.132070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.132037    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:19.506373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.506305    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"]
Nov 03 12:31:19.515538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.515513    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"
Nov 03 12:31:19.515673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.515532    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb" event={"ID":"84c81586-3b88-418d-a3f5-742caf20efe4","Type":"ContainerDied","Data":"39601fc02537b74789bee1e3a6a1d87d96a799d71156873cdf350c62d6dca431"}
Nov 03 12:31:19.515673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.515591    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39601fc02537b74789bee1e3a6a1d87d96a799d71156873cdf350c62d6dca431"
Nov 03 12:31:19.775639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.775558    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"]
Nov 03 12:31:19.776069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.775855    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv" podUID="14e9dcc1-03d9-4784-a732-ef63aaf95c40" containerName="payload-processing" containerID="cri-o://c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53" gracePeriod=30
Nov 03 12:31:19.828564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.828532    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"]
Nov 03 12:31:19.828837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:19.828792    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq" podUID="f33427bb-f815-41e9-8e3c-97b172353553" containerName="payload-pre-processing" containerID="cri-o://3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f" gracePeriod=30
Nov 03 12:31:20.130392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.130362    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:31:20.132544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.132521    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:31:20.326370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.325810    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x\" (UniqueName: \"kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x\") pod \"f33427bb-f815-41e9-8e3c-97b172353553\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") "
Nov 03 12:31:20.326370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.326361    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume\") pod \"f33427bb-f815-41e9-8e3c-97b172353553\" (UID: \"f33427bb-f815-41e9-8e3c-97b172353553\") "
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.326999    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume" pod "f33427bb-f815-41e9-8e3c-97b172353553" (UID: "f33427bb-f815-41e9-8e3c-97b172353553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.327391    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume\") pod \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") "
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.327438    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6\" (UniqueName: \"kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6\") pod \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\" (UID: \"14e9dcc1-03d9-4784-a732-ef63aaf95c40\") "
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.328027    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33427bb-f815-41e9-8e3c-97b172353553-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.328647    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume" pod "14e9dcc1-03d9-4784-a732-ef63aaf95c40" (UID: "14e9dcc1-03d9-4784-a732-ef63aaf95c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.330280    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6" pod "14e9dcc1-03d9-4784-a732-ef63aaf95c40" (UID: "14e9dcc1-03d9-4784-a732-ef63aaf95c40"). InnerVolumeSpecName "kube-api-access-w6sz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:20.332681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.332141    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x" pod "f33427bb-f815-41e9-8e3c-97b172353553" (UID: "f33427bb-f815-41e9-8e3c-97b172353553"). InnerVolumeSpecName "kube-api-access-b525x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:20.429486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.429442    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e9dcc1-03d9-4784-a732-ef63aaf95c40-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:20.429486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.429479    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6sz6\" (UniqueName: \"kubernetes.io/projected/14e9dcc1-03d9-4784-a732-ef63aaf95c40-kube-api-access-w6sz6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:20.429486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.429491    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b525x\" (UniqueName: \"kubernetes.io/projected/f33427bb-f815-41e9-8e3c-97b172353553-kube-api-access-b525x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:20.522878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.522841    4546 generic.go:356] "Generic (PLEG): container finished" podID="37b9efbf-3414-4bdc-b7de-bc53af0e1963" containerID="2561cab92d7e95d403aa2d187b0449ba12b7f9ba5cb554b8bafbc6bf96444216" exitCode=0
Nov 03 12:31:20.523077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.522927    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q" event={"ID":"37b9efbf-3414-4bdc-b7de-bc53af0e1963","Type":"ContainerDied","Data":"2561cab92d7e95d403aa2d187b0449ba12b7f9ba5cb554b8bafbc6bf96444216"}
Nov 03 12:31:20.523077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.522990    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q" event={"ID":"37b9efbf-3414-4bdc-b7de-bc53af0e1963","Type":"ContainerStarted","Data":"627fc6f293bda9e85705fba2d3ddc564273a501d4653ee8ea7aeb3af5780e905"}
Nov 03 12:31:20.524255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.524235    4546 generic.go:356] "Generic (PLEG): container finished" podID="14e9dcc1-03d9-4784-a732-ef63aaf95c40" containerID="c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53" exitCode=0
Nov 03 12:31:20.524364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.524273    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"
Nov 03 12:31:20.524364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.524305    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv" event={"ID":"14e9dcc1-03d9-4784-a732-ef63aaf95c40","Type":"ContainerDied","Data":"c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53"}
Nov 03 12:31:20.524364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.524338    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv" event={"ID":"14e9dcc1-03d9-4784-a732-ef63aaf95c40","Type":"ContainerDied","Data":"927cba1ed315c524714dadbe24db8d06e2753e2ccff96c6bba2ec6ab0f3de6be"}
Nov 03 12:31:20.524507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.524366    4546 scope.go:122] "RemoveContainer" containerID="c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53"
Nov 03 12:31:20.525691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.525671    4546 generic.go:356] "Generic (PLEG): container finished" podID="f33427bb-f815-41e9-8e3c-97b172353553" containerID="3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f" exitCode=0
Nov 03 12:31:20.525813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.525714    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"
Nov 03 12:31:20.525813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.525760    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq" event={"ID":"f33427bb-f815-41e9-8e3c-97b172353553","Type":"ContainerDied","Data":"3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f"}
Nov 03 12:31:20.525813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.525800    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq" event={"ID":"f33427bb-f815-41e9-8e3c-97b172353553","Type":"ContainerDied","Data":"0d45056612d9a290550c917c5daadf63aed19639a37b001fec42432b44a4a1bc"}
Nov 03 12:31:20.573004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.572872    4546 scope.go:122] "RemoveContainer" containerID="c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53"
Nov 03 12:31:20.573200 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:20.573179    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53\": container with ID starting with c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53 not found: ID does not exist" containerID="c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53"
Nov 03 12:31:20.573263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.573209    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53"} err="failed to get container status \"c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53\": rpc error: code = NotFound desc = could not find container \"c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53\": container with ID starting with c68810e501751992907c1f41e017048139f18b847bd37133c60cef92f8076b53 not found: ID does not exist"
Nov 03 12:31:20.573263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.573226    4546 scope.go:122] "RemoveContainer" containerID="3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f"
Nov 03 12:31:20.583869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.583849    4546 scope.go:122] "RemoveContainer" containerID="3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f"
Nov 03 12:31:20.584145 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:20.584124    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f\": container with ID starting with 3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f not found: ID does not exist" containerID="3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f"
Nov 03 12:31:20.584194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.584153    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f"} err="failed to get container status \"3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f\": rpc error: code = NotFound desc = could not find container \"3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f\": container with ID starting with 3dde41e0e3dd2bb76b02f0b5bdbb3061408acfe17ce28f376f2014aca0e6104f not found: ID does not exist"
Nov 03 12:31:20.732323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.732285    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"]
Nov 03 12:31:20.738290 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.738256    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d25504-64b54fcdd4-4hlkv"]
Nov 03 12:31:20.838800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.838691    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"]
Nov 03 12:31:20.842035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:20.842010    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d25504-7f9c4dd6ff-4m5nq"]
Nov 03 12:31:21.679583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.679554    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:21.744589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.744560    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca\") pod \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") "
Nov 03 12:31:21.744854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.744615    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca\") pod \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") "
Nov 03 12:31:21.744854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.744649    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg\" (UniqueName: \"kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg\") pod \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\" (UID: \"37b9efbf-3414-4bdc-b7de-bc53af0e1963\") "
Nov 03 12:31:21.745089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.745055    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca" pod "37b9efbf-3414-4bdc-b7de-bc53af0e1963" (UID: "37b9efbf-3414-4bdc-b7de-bc53af0e1963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:21.747018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.746987    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca" pod "37b9efbf-3414-4bdc-b7de-bc53af0e1963" (UID: "37b9efbf-3414-4bdc-b7de-bc53af0e1963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:21.747105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.747072    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg" pod "37b9efbf-3414-4bdc-b7de-bc53af0e1963" (UID: "37b9efbf-3414-4bdc-b7de-bc53af0e1963"). InnerVolumeSpecName "kube-api-access-zzgwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:21.846520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.846427    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:21.846520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.846462    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b9efbf-3414-4bdc-b7de-bc53af0e1963-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:21.846520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:21.846474    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgwg\" (UniqueName: \"kubernetes.io/projected/37b9efbf-3414-4bdc-b7de-bc53af0e1963-kube-api-access-zzgwg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:22.541566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.541533    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"
Nov 03 12:31:22.541566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.541545    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q" event={"ID":"37b9efbf-3414-4bdc-b7de-bc53af0e1963","Type":"ContainerDied","Data":"627fc6f293bda9e85705fba2d3ddc564273a501d4653ee8ea7aeb3af5780e905"}
Nov 03 12:31:22.541829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.541641    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="627fc6f293bda9e85705fba2d3ddc564273a501d4653ee8ea7aeb3af5780e905"
Nov 03 12:31:22.573295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.573264    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e9dcc1-03d9-4784-a732-ef63aaf95c40" path="/var/lib/kubelet/pods/14e9dcc1-03d9-4784-a732-ef63aaf95c40/volumes"
Nov 03 12:31:22.573668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.573655    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33427bb-f815-41e9-8e3c-97b172353553" path="/var/lib/kubelet/pods/f33427bb-f815-41e9-8e3c-97b172353553/volumes"
Nov 03 12:31:22.746135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746098    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"]
Nov 03 12:31:22.746688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746668    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f33427bb-f815-41e9-8e3c-97b172353553" containerName="payload-pre-processing"
Nov 03 12:31:22.746688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746685    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33427bb-f815-41e9-8e3c-97b172353553" containerName="payload-pre-processing"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746699    4546 cpu_manager.go:414] "containerMap: removing container" podUID="14e9dcc1-03d9-4784-a732-ef63aaf95c40" containerName="payload-processing"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746707    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e9dcc1-03d9-4784-a732-ef63aaf95c40" containerName="payload-processing"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746725    4546 cpu_manager.go:414] "containerMap: removing container" podUID="37b9efbf-3414-4bdc-b7de-bc53af0e1963" containerName="revoke-keys"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746732    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b9efbf-3414-4bdc-b7de-bc53af0e1963" containerName="revoke-keys"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746834    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33427bb-f815-41e9-8e3c-97b172353553" containerName="payload-pre-processing"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746844    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e9dcc1-03d9-4784-a732-ef63aaf95c40" containerName="payload-processing"
Nov 03 12:31:22.746918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.746852    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b9efbf-3414-4bdc-b7de-bc53af0e1963" containerName="revoke-keys"
Nov 03 12:31:22.750718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.750698    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.803586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.803169    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:31:22.803586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.803504    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:31:22.804625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.804596    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"]
Nov 03 12:31:22.846251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.846218    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:31:22.846429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.846278    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:31:22.865907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.865871    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.866289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.865932    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.866289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.866033    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98gj\" (UniqueName: \"kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.967252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.967211    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.967441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.967278    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.967441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.967329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98gj\" (UniqueName: \"kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.967997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.967978    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:22.969819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:22.969800    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:23.023554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:23.023512    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98gj\" (UniqueName: \"kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj\") pod \"maas-api-revoke-sub-dc9ede3e-tst6m\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:23.066058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:23.065977    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:23.435492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:23.435462    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"]
Nov 03 12:31:23.435642 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:23.435467    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2decc78_a8ab_4363_94a0_b5270d18d3e8.slice/crio-95499e57ab2768cb391be68d27dba2c08763b674a2d49e8d6b95c14392187a48 WatchSource:0}: Error finding container 95499e57ab2768cb391be68d27dba2c08763b674a2d49e8d6b95c14392187a48: Status 404 returned error can't find the container with id 95499e57ab2768cb391be68d27dba2c08763b674a2d49e8d6b95c14392187a48
Nov 03 12:31:23.548660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:23.548627    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m" event={"ID":"b2decc78-a8ab-4363-94a0-b5270d18d3e8","Type":"ContainerStarted","Data":"95499e57ab2768cb391be68d27dba2c08763b674a2d49e8d6b95c14392187a48"}
Nov 03 12:31:24.557126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:24.557033    4546 generic.go:356] "Generic (PLEG): container finished" podID="b2decc78-a8ab-4363-94a0-b5270d18d3e8" containerID="87f64dd4e08c320b389ec94f6498fc4b1e1e4fccdced0c41be0697016d1f5583" exitCode=0
Nov 03 12:31:24.557593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:24.557112    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m" event={"ID":"b2decc78-a8ab-4363-94a0-b5270d18d3e8","Type":"ContainerDied","Data":"87f64dd4e08c320b389ec94f6498fc4b1e1e4fccdced0c41be0697016d1f5583"}
Nov 03 12:31:25.736656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.736626    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:25.801786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.801738    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca\") pod \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") "
Nov 03 12:31:25.801964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.801935    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj\" (UniqueName: \"kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj\") pod \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") "
Nov 03 12:31:25.802065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.802047    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca\") pod \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\" (UID: \"b2decc78-a8ab-4363-94a0-b5270d18d3e8\") "
Nov 03 12:31:25.802187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.802158    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca" pod "b2decc78-a8ab-4363-94a0-b5270d18d3e8" (UID: "b2decc78-a8ab-4363-94a0-b5270d18d3e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:25.802791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.802771    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:25.804165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.804142    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj" pod "b2decc78-a8ab-4363-94a0-b5270d18d3e8" (UID: "b2decc78-a8ab-4363-94a0-b5270d18d3e8"). InnerVolumeSpecName "kube-api-access-h98gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:25.804251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.804233    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca" pod "b2decc78-a8ab-4363-94a0-b5270d18d3e8" (UID: "b2decc78-a8ab-4363-94a0-b5270d18d3e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:25.904456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.904353    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98gj\" (UniqueName: \"kubernetes.io/projected/b2decc78-a8ab-4363-94a0-b5270d18d3e8-kube-api-access-h98gj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:25.904456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:25.904402    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2decc78-a8ab-4363-94a0-b5270d18d3e8-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:26.571836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:26.571811    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"
Nov 03 12:31:26.573822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:26.573790    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m" event={"ID":"b2decc78-a8ab-4363-94a0-b5270d18d3e8","Type":"ContainerDied","Data":"95499e57ab2768cb391be68d27dba2c08763b674a2d49e8d6b95c14392187a48"}
Nov 03 12:31:26.573976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:26.573837    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95499e57ab2768cb391be68d27dba2c08763b674a2d49e8d6b95c14392187a48"
Nov 03 12:31:28.195823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.195789    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"]
Nov 03 12:31:28.196621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.196594    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b2decc78-a8ab-4363-94a0-b5270d18d3e8" containerName="revoke-keys"
Nov 03 12:31:28.196621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.196621    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2decc78-a8ab-4363-94a0-b5270d18d3e8" containerName="revoke-keys"
Nov 03 12:31:28.196821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.196768    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2decc78-a8ab-4363-94a0-b5270d18d3e8" containerName="revoke-keys"
Nov 03 12:31:28.200400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.200379    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.295392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.295328    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:31:28.295392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.295354    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:31:28.295392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.295346    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:31:28.295768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.295417    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:31:28.299469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.299446    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"]
Nov 03 12:31:28.340890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.340854    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.341068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.340900    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhtq\" (UniqueName: \"kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.341068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.341053    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.442433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.442400    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.442645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.442445    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhtq\" (UniqueName: \"kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.442645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.442514    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.443301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.443229    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.445038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.445016    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.495446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.495415    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhtq\" (UniqueName: \"kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq\") pod \"maas-api-revoke-sub-aab008ca-sm4f5\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.516975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.516938    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:28.896673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:28.896640    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"]
Nov 03 12:31:28.897003 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:28.896978    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3db6af_6c64_43bd_99cf_f26d937313b8.slice/crio-dc14df4a6564f37be521336d9d9c52d0077d53bcd09249808d93e09bc60d83a3 WatchSource:0}: Error finding container dc14df4a6564f37be521336d9d9c52d0077d53bcd09249808d93e09bc60d83a3: Status 404 returned error can't find the container with id dc14df4a6564f37be521336d9d9c52d0077d53bcd09249808d93e09bc60d83a3
Nov 03 12:31:29.607883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:29.607776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5" event={"ID":"ce3db6af-6c64-43bd-99cf-f26d937313b8","Type":"ContainerStarted","Data":"324ec87d3ed4ae368af9dd13d788506f77b5957bb3f2260dece85bf356cc7a78"}
Nov 03 12:31:29.607883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:29.607848    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5" event={"ID":"ce3db6af-6c64-43bd-99cf-f26d937313b8","Type":"ContainerStarted","Data":"dc14df4a6564f37be521336d9d9c52d0077d53bcd09249808d93e09bc60d83a3"}
Nov 03 12:31:29.716531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:29.716474    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5" podStartSLOduration=1.716458241 podStartE2EDuration="1.716458241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:29.712252343 +0000 UTC m=+2431.283836294" watchObservedRunningTime="2026-11-03 12:31:29.716458241 +0000 UTC m=+2431.288042168"
Nov 03 12:31:30.652776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:30.652465    4546 generic.go:356] "Generic (PLEG): container finished" podID="ce3db6af-6c64-43bd-99cf-f26d937313b8" containerID="324ec87d3ed4ae368af9dd13d788506f77b5957bb3f2260dece85bf356cc7a78" exitCode=0
Nov 03 12:31:30.652776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:30.652722    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5" event={"ID":"ce3db6af-6c64-43bd-99cf-f26d937313b8","Type":"ContainerDied","Data":"324ec87d3ed4ae368af9dd13d788506f77b5957bb3f2260dece85bf356cc7a78"}
Nov 03 12:31:31.811536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.811505    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:31.915619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.915522    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq\" (UniqueName: \"kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq\") pod \"ce3db6af-6c64-43bd-99cf-f26d937313b8\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") "
Nov 03 12:31:31.915819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.915641    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca\") pod \"ce3db6af-6c64-43bd-99cf-f26d937313b8\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") "
Nov 03 12:31:31.915819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.915710    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca\") pod \"ce3db6af-6c64-43bd-99cf-f26d937313b8\" (UID: \"ce3db6af-6c64-43bd-99cf-f26d937313b8\") "
Nov 03 12:31:31.916089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.916053    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca" pod "ce3db6af-6c64-43bd-99cf-f26d937313b8" (UID: "ce3db6af-6c64-43bd-99cf-f26d937313b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:31.916246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.916231    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:31.918188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.918161    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca" pod "ce3db6af-6c64-43bd-99cf-f26d937313b8" (UID: "ce3db6af-6c64-43bd-99cf-f26d937313b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:31.918188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:31.918167    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq" pod "ce3db6af-6c64-43bd-99cf-f26d937313b8" (UID: "ce3db6af-6c64-43bd-99cf-f26d937313b8"). InnerVolumeSpecName "kube-api-access-glhtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:32.017407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:32.017365    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhtq\" (UniqueName: \"kubernetes.io/projected/ce3db6af-6c64-43bd-99cf-f26d937313b8-kube-api-access-glhtq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:32.017407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:32.017397    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3db6af-6c64-43bd-99cf-f26d937313b8-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:32.674972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:32.674939    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"
Nov 03 12:31:32.675137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:32.674969    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5" event={"ID":"ce3db6af-6c64-43bd-99cf-f26d937313b8","Type":"ContainerDied","Data":"dc14df4a6564f37be521336d9d9c52d0077d53bcd09249808d93e09bc60d83a3"}
Nov 03 12:31:32.675137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:32.675036    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc14df4a6564f37be521336d9d9c52d0077d53bcd09249808d93e09bc60d83a3"
Nov 03 12:31:37.654351 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:37.654309    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:31:38.728297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.728266    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"]
Nov 03 12:31:38.728869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.728852    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ce3db6af-6c64-43bd-99cf-f26d937313b8" containerName="revoke-keys"
Nov 03 12:31:38.728942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.728869    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3db6af-6c64-43bd-99cf-f26d937313b8" containerName="revoke-keys"
Nov 03 12:31:38.728991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.728972    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3db6af-6c64-43bd-99cf-f26d937313b8" containerName="revoke-keys"
Nov 03 12:31:38.732640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.732625    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.814506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.814466    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b529k\" (UniqueName: \"kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.814696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.814659    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.814798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.814701    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.828033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.828005    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:31:38.828033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.828017    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:31:38.828211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.828052    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:31:38.828211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.828084    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:31:38.831811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.831789    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"]
Nov 03 12:31:38.889833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.889805    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"]
Nov 03 12:31:38.915869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.915833    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b529k\" (UniqueName: \"kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.916037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.915950    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.916037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.915992    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.916645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.916621    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.918375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.918353    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.968400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.968369    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b529k\" (UniqueName: \"kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k\") pod \"maas-api-revoke-sub-b198df22-66hwq\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:38.989535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:38.989464    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"]
Nov 03 12:31:39.045586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:39.045559    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:39.411765 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:39.411710    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"]
Nov 03 12:31:39.413084 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:39.413053    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267fdba9_1d5d_41ce_9abd_c306d457358c.slice/crio-7ae9d7a0f620df46bfcce1ff160952e0e39d5c5b956cdebeeb5dfc74359bffc6 WatchSource:0}: Error finding container 7ae9d7a0f620df46bfcce1ff160952e0e39d5c5b956cdebeeb5dfc74359bffc6: Status 404 returned error can't find the container with id 7ae9d7a0f620df46bfcce1ff160952e0e39d5c5b956cdebeeb5dfc74359bffc6
Nov 03 12:31:39.724105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:39.724073    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq" event={"ID":"267fdba9-1d5d-41ce-9abd-c306d457358c","Type":"ContainerStarted","Data":"7ae9d7a0f620df46bfcce1ff160952e0e39d5c5b956cdebeeb5dfc74359bffc6"}
Nov 03 12:31:40.732487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:40.732456    4546 generic.go:356] "Generic (PLEG): container finished" podID="267fdba9-1d5d-41ce-9abd-c306d457358c" containerID="0fe173731c36289a78fc5da3c7f41c5430d38cdf2a7f7155a4b39f9f7fc17b4c" exitCode=0
Nov 03 12:31:40.732881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:40.732545    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq" event={"ID":"267fdba9-1d5d-41ce-9abd-c306d457358c","Type":"ContainerDied","Data":"0fe173731c36289a78fc5da3c7f41c5430d38cdf2a7f7155a4b39f9f7fc17b4c"}
Nov 03 12:31:41.883153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.883127    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:41.960785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.960725    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k\" (UniqueName: \"kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k\") pod \"267fdba9-1d5d-41ce-9abd-c306d457358c\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") "
Nov 03 12:31:41.960963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.960795    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca\") pod \"267fdba9-1d5d-41ce-9abd-c306d457358c\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") "
Nov 03 12:31:41.960963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.960877    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca\") pod \"267fdba9-1d5d-41ce-9abd-c306d457358c\" (UID: \"267fdba9-1d5d-41ce-9abd-c306d457358c\") "
Nov 03 12:31:41.961261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.961235    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca" pod "267fdba9-1d5d-41ce-9abd-c306d457358c" (UID: "267fdba9-1d5d-41ce-9abd-c306d457358c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:41.961388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.961372    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:41.962994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.962962    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k" pod "267fdba9-1d5d-41ce-9abd-c306d457358c" (UID: "267fdba9-1d5d-41ce-9abd-c306d457358c"). InnerVolumeSpecName "kube-api-access-b529k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:41.963102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:41.962997    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca" pod "267fdba9-1d5d-41ce-9abd-c306d457358c" (UID: "267fdba9-1d5d-41ce-9abd-c306d457358c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:42.062345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:42.062252    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b529k\" (UniqueName: \"kubernetes.io/projected/267fdba9-1d5d-41ce-9abd-c306d457358c-kube-api-access-b529k\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:42.062345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:42.062288    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267fdba9-1d5d-41ce-9abd-c306d457358c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:42.746344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:42.746307    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"
Nov 03 12:31:42.746561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:42.746342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq" event={"ID":"267fdba9-1d5d-41ce-9abd-c306d457358c","Type":"ContainerDied","Data":"7ae9d7a0f620df46bfcce1ff160952e0e39d5c5b956cdebeeb5dfc74359bffc6"}
Nov 03 12:31:42.746561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:42.746391    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae9d7a0f620df46bfcce1ff160952e0e39d5c5b956cdebeeb5dfc74359bffc6"
Nov 03 12:31:43.890278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:43.890233    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7" podUID="13240456-69cd-444c-bffd-863a8e8f2bae" containerName="payload-processing" containerID="cri-o://e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3" gracePeriod=25
Nov 03 12:31:43.997855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:43.989919    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp" podUID="0e25fd51-8258-41b1-a7f5-19362f9af42f" containerName="payload-pre-processing" containerID="cri-o://82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7" gracePeriod=25
Nov 03 12:31:44.229801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.228808    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:31:44.229801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.229448    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"]
Nov 03 12:31:44.229801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.229690    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" podUID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" containerName="payload-processing" containerID="cri-o://16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889" gracePeriod=30
Nov 03 12:31:44.275145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.275104    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"]
Nov 03 12:31:44.275872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.275430    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq" podUID="6f3633a9-8fc6-4d8d-9318-60089e2a56ac" containerName="payload-pre-processing" containerID="cri-o://9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2" gracePeriod=30
Nov 03 12:31:44.277914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.277883    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" podUID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.200:9004: connect: connection refused"
Nov 03 12:31:44.392411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.392385    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:31:44.401101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.401072    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume\") pod \"13240456-69cd-444c-bffd-863a8e8f2bae\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") "
Nov 03 12:31:44.401275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.401238    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr\" (UniqueName: \"kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr\") pod \"13240456-69cd-444c-bffd-863a8e8f2bae\" (UID: \"13240456-69cd-444c-bffd-863a8e8f2bae\") "
Nov 03 12:31:44.402525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.402495    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume" pod "13240456-69cd-444c-bffd-863a8e8f2bae" (UID: "13240456-69cd-444c-bffd-863a8e8f2bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:44.405631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.405583    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr" pod "13240456-69cd-444c-bffd-863a8e8f2bae" (UID: "13240456-69cd-444c-bffd-863a8e8f2bae"). InnerVolumeSpecName "kube-api-access-mc2fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:44.502297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.502266    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w\" (UniqueName: \"kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w\") pod \"0e25fd51-8258-41b1-a7f5-19362f9af42f\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") "
Nov 03 12:31:44.502471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.502453    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume\") pod \"0e25fd51-8258-41b1-a7f5-19362f9af42f\" (UID: \"0e25fd51-8258-41b1-a7f5-19362f9af42f\") "
Nov 03 12:31:44.502922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.502897    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13240456-69cd-444c-bffd-863a8e8f2bae-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.503065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.502926    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc2fr\" (UniqueName: \"kubernetes.io/projected/13240456-69cd-444c-bffd-863a8e8f2bae-kube-api-access-mc2fr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.503132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.503075    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume" pod "0e25fd51-8258-41b1-a7f5-19362f9af42f" (UID: "0e25fd51-8258-41b1-a7f5-19362f9af42f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:44.509502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.509465    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w" pod "0e25fd51-8258-41b1-a7f5-19362f9af42f" (UID: "0e25fd51-8258-41b1-a7f5-19362f9af42f"). InnerVolumeSpecName "kube-api-access-gpw9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:44.546270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.546239    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:31:44.566736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.566707    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:31:44.604125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.604101    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpw9w\" (UniqueName: \"kubernetes.io/projected/0e25fd51-8258-41b1-a7f5-19362f9af42f-kube-api-access-gpw9w\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.604125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.604126    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e25fd51-8258-41b1-a7f5-19362f9af42f-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.704779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.704667    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume\") pod \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") "
Nov 03 12:31:44.704980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.704786    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume\") pod \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") "
Nov 03 12:31:44.704980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.704959    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf\" (UniqueName: \"kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf\") pod \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\" (UID: \"6f3633a9-8fc6-4d8d-9318-60089e2a56ac\") "
Nov 03 12:31:44.705123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.705041    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr\" (UniqueName: \"kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr\") pod \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\" (UID: \"96a78d15-f4aa-40af-a0a2-42fe843a6a6b\") "
Nov 03 12:31:44.705188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.705142    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume" pod "96a78d15-f4aa-40af-a0a2-42fe843a6a6b" (UID: "96a78d15-f4aa-40af-a0a2-42fe843a6a6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:44.705242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.705217    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume" pod "6f3633a9-8fc6-4d8d-9318-60089e2a56ac" (UID: "6f3633a9-8fc6-4d8d-9318-60089e2a56ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:44.705605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.705583    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.705605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.705608    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.707284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.707255    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr" pod "96a78d15-f4aa-40af-a0a2-42fe843a6a6b" (UID: "96a78d15-f4aa-40af-a0a2-42fe843a6a6b"). InnerVolumeSpecName "kube-api-access-7k8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:44.707371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.707338    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf" pod "6f3633a9-8fc6-4d8d-9318-60089e2a56ac" (UID: "6f3633a9-8fc6-4d8d-9318-60089e2a56ac"). InnerVolumeSpecName "kube-api-access-8qgkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:44.763507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.763476    4546 generic.go:356] "Generic (PLEG): container finished" podID="13240456-69cd-444c-bffd-863a8e8f2bae" containerID="e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3" exitCode=0
Nov 03 12:31:44.763694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.763542    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"
Nov 03 12:31:44.763694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.763554    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7" event={"ID":"13240456-69cd-444c-bffd-863a8e8f2bae","Type":"ContainerDied","Data":"e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3"}
Nov 03 12:31:44.763694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.763613    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7" event={"ID":"13240456-69cd-444c-bffd-863a8e8f2bae","Type":"ContainerDied","Data":"61a2c3869d1b82e9ec6af7f9d19e32f6605f02946308fb1afd11e0c345e6b1ee"}
Nov 03 12:31:44.763694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.763644    4546 scope.go:122] "RemoveContainer" containerID="e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3"
Nov 03 12:31:44.765421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.765399    4546 generic.go:356] "Generic (PLEG): container finished" podID="6f3633a9-8fc6-4d8d-9318-60089e2a56ac" containerID="9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2" exitCode=0
Nov 03 12:31:44.765561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.765445    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"
Nov 03 12:31:44.765561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.765491    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq" event={"ID":"6f3633a9-8fc6-4d8d-9318-60089e2a56ac","Type":"ContainerDied","Data":"9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2"}
Nov 03 12:31:44.765561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.765545    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq" event={"ID":"6f3633a9-8fc6-4d8d-9318-60089e2a56ac","Type":"ContainerDied","Data":"63f41e4a67f3c8de6dbeb06cb1e2a2bcf73676452ff178c1cb55a5b7caccf75b"}
Nov 03 12:31:44.767384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.767366    4546 generic.go:356] "Generic (PLEG): container finished" podID="0e25fd51-8258-41b1-a7f5-19362f9af42f" containerID="82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7" exitCode=0
Nov 03 12:31:44.767496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.767483    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"
Nov 03 12:31:44.767589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.767561    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp" event={"ID":"0e25fd51-8258-41b1-a7f5-19362f9af42f","Type":"ContainerDied","Data":"82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7"}
Nov 03 12:31:44.767732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.767714    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp" event={"ID":"0e25fd51-8258-41b1-a7f5-19362f9af42f","Type":"ContainerDied","Data":"6f82c6784350932ab0f27385ca3b08d208a601de98df1dde214e379de3a247a7"}
Nov 03 12:31:44.769254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.769231    4546 generic.go:356] "Generic (PLEG): container finished" podID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" containerID="16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889" exitCode=0
Nov 03 12:31:44.769359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.769267    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" event={"ID":"96a78d15-f4aa-40af-a0a2-42fe843a6a6b","Type":"ContainerDied","Data":"16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889"}
Nov 03 12:31:44.769359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.769296    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87" event={"ID":"96a78d15-f4aa-40af-a0a2-42fe843a6a6b","Type":"ContainerDied","Data":"28ffe85c834d67530e0f24d1e4245f9a97a7c88db4604afcc1ec7192336035b5"}
Nov 03 12:31:44.769359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.769347    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"
Nov 03 12:31:44.778550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.778534    4546 scope.go:122] "RemoveContainer" containerID="e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3"
Nov 03 12:31:44.778981 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:44.778878    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3\": container with ID starting with e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3 not found: ID does not exist" containerID="e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3"
Nov 03 12:31:44.778981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.778910    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3"} err="failed to get container status \"e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3\": rpc error: code = NotFound desc = could not find container \"e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3\": container with ID starting with e4ff306f4317a8ae05b92d3d29f3eae93edbc258ac7a81abf90198111553b8d3 not found: ID does not exist"
Nov 03 12:31:44.778981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.778933    4546 scope.go:122] "RemoveContainer" containerID="9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2"
Nov 03 12:31:44.793702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.793679    4546 scope.go:122] "RemoveContainer" containerID="9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2"
Nov 03 12:31:44.794463 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:44.794439    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2\": container with ID starting with 9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2 not found: ID does not exist" containerID="9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2"
Nov 03 12:31:44.794646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.794617    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2"} err="failed to get container status \"9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2\": rpc error: code = NotFound desc = could not find container \"9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2\": container with ID starting with 9aa53b3b4bf82b67454f9beb4ac596bc66e0707f720b756e3a8a080649aec8e2 not found: ID does not exist"
Nov 03 12:31:44.794799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.794781    4546 scope.go:122] "RemoveContainer" containerID="82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7"
Nov 03 12:31:44.820408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.820378    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgkf\" (UniqueName: \"kubernetes.io/projected/6f3633a9-8fc6-4d8d-9318-60089e2a56ac-kube-api-access-8qgkf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.820602 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.820576    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8tr\" (UniqueName: \"kubernetes.io/projected/96a78d15-f4aa-40af-a0a2-42fe843a6a6b-kube-api-access-7k8tr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.856787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.856456    4546 scope.go:122] "RemoveContainer" containerID="82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7"
Nov 03 12:31:44.857798 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:44.857359    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7\": container with ID starting with 82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7 not found: ID does not exist" containerID="82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7"
Nov 03 12:31:44.857798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.857400    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7"} err="failed to get container status \"82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7\": rpc error: code = NotFound desc = could not find container \"82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7\": container with ID starting with 82121a6a69ded2e528a298bf23b3b8c31a21a6a19206e60d44aa9867f3c142b7 not found: ID does not exist"
Nov 03 12:31:44.857798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.857428    4546 scope.go:122] "RemoveContainer" containerID="16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889"
Nov 03 12:31:44.868557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.868537    4546 scope.go:122] "RemoveContainer" containerID="16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889"
Nov 03 12:31:44.868865 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:44.868841    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889\": container with ID starting with 16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889 not found: ID does not exist" containerID="16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889"
Nov 03 12:31:44.868970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.868876    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889"} err="failed to get container status \"16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889\": rpc error: code = NotFound desc = could not find container \"16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889\": container with ID starting with 16156040c4823b299776f1ff4db9e0f9011c1bdc2616591ee13b1719e1b8f889 not found: ID does not exist"
Nov 03 12:31:44.981731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.981696    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"]
Nov 03 12:31:44.987077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:44.987042    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c02ae4-9f59f6c9-lsrjq"]
Nov 03 12:31:45.089283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:45.089254    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"]
Nov 03 12:31:45.092727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:45.092704    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-5ccd7d94ff-kpwb7"]
Nov 03 12:31:45.192556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:45.192526    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"]
Nov 03 12:31:45.198392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:45.198367    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-778db65795-p5fgp"]
Nov 03 12:31:45.297772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:45.297691    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"]
Nov 03 12:31:45.303659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:45.303629    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c02ae4-66747b9576-27b87"]
Nov 03 12:31:46.573129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:46.573099    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e25fd51-8258-41b1-a7f5-19362f9af42f" path="/var/lib/kubelet/pods/0e25fd51-8258-41b1-a7f5-19362f9af42f/volumes"
Nov 03 12:31:46.573525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:46.573510    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13240456-69cd-444c-bffd-863a8e8f2bae" path="/var/lib/kubelet/pods/13240456-69cd-444c-bffd-863a8e8f2bae/volumes"
Nov 03 12:31:46.573881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:46.573868    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3633a9-8fc6-4d8d-9318-60089e2a56ac" path="/var/lib/kubelet/pods/6f3633a9-8fc6-4d8d-9318-60089e2a56ac/volumes"
Nov 03 12:31:46.574212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:46.574201    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" path="/var/lib/kubelet/pods/96a78d15-f4aa-40af-a0a2-42fe843a6a6b/volumes"
Nov 03 12:31:50.936797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.936763    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"]
Nov 03 12:31:50.937408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.937038    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" containerID="cri-o://7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b" gracePeriod=30
Nov 03 12:31:50.957502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.957467    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"]
Nov 03 12:31:50.957996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.957981    4546 cpu_manager.go:414] "containerMap: removing container" podUID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" containerName="payload-processing"
Nov 03 12:31:50.957996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.957996    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" containerName="payload-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958006    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6f3633a9-8fc6-4d8d-9318-60089e2a56ac" containerName="payload-pre-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958012    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3633a9-8fc6-4d8d-9318-60089e2a56ac" containerName="payload-pre-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958023    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0e25fd51-8258-41b1-a7f5-19362f9af42f" containerName="payload-pre-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958028    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e25fd51-8258-41b1-a7f5-19362f9af42f" containerName="payload-pre-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958049    4546 cpu_manager.go:414] "containerMap: removing container" podUID="13240456-69cd-444c-bffd-863a8e8f2bae" containerName="payload-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958053    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13240456-69cd-444c-bffd-863a8e8f2bae" containerName="payload-processing"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958065    4546 cpu_manager.go:414] "containerMap: removing container" podUID="267fdba9-1d5d-41ce-9abd-c306d457358c" containerName="revoke-keys"
Nov 03 12:31:50.958131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958069    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267fdba9-1d5d-41ce-9abd-c306d457358c" containerName="revoke-keys"
Nov 03 12:31:50.958383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958164    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="13240456-69cd-444c-bffd-863a8e8f2bae" containerName="payload-processing"
Nov 03 12:31:50.958383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958175    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="267fdba9-1d5d-41ce-9abd-c306d457358c" containerName="revoke-keys"
Nov 03 12:31:50.958383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958185    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a78d15-f4aa-40af-a0a2-42fe843a6a6b" containerName="payload-processing"
Nov 03 12:31:50.958383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958193    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3633a9-8fc6-4d8d-9318-60089e2a56ac" containerName="payload-pre-processing"
Nov 03 12:31:50.958383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.958198    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e25fd51-8258-41b1-a7f5-19362f9af42f" containerName="payload-pre-processing"
Nov 03 12:31:50.965789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.965637    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:50.993668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.993635    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:50.993851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.993683    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:50.993851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:50.993784    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9jr\" (UniqueName: \"kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.016277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.016242    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:31:51.016492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.016470    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"]
Nov 03 12:31:51.020708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.020673    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:31:51.095423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.094054    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:31:51.095423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.094176    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:31:51.095423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.094627    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.095423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.094694    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.095423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.094773    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9jr\" (UniqueName: \"kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.095864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.095630    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.105810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.098616    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.147597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.147569    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9jr\" (UniqueName: \"kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr\") pod \"maas-api-revoke-sub-6306015d-rfx2z\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.280657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.280628    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:51.647660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.647630    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"]
Nov 03 12:31:51.649067 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:51.649038    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4af4b40_a245_4146_b652_dd9333b0a4eb.slice/crio-1b6f0b5fe4e409a1d0ad06b62d470b6a33e98da6be9d0e253f5e777f324963d2 WatchSource:0}: Error finding container 1b6f0b5fe4e409a1d0ad06b62d470b6a33e98da6be9d0e253f5e777f324963d2: Status 404 returned error can't find the container with id 1b6f0b5fe4e409a1d0ad06b62d470b6a33e98da6be9d0e253f5e777f324963d2
Nov 03 12:31:51.835794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:51.835738    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z" event={"ID":"a4af4b40-a245-4146-b652-dd9333b0a4eb","Type":"ContainerStarted","Data":"1b6f0b5fe4e409a1d0ad06b62d470b6a33e98da6be9d0e253f5e777f324963d2"}
Nov 03 12:31:52.158128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:52.158094    4546 patch_prober.go:29] interesting pod/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:31:52.158520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:52.158147    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:31:52.522960 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:52.522924    4546 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4af4b40_a245_4146_b652_dd9333b0a4eb.slice/crio-b6ceba5c65debeece812131e20ac5b8980bcf985cc625dac875dbba96c04d6c4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4af4b40_a245_4146_b652_dd9333b0a4eb.slice/crio-conmon-b6ceba5c65debeece812131e20ac5b8980bcf985cc625dac875dbba96c04d6c4.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 12:31:52.843470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:52.843375    4546 generic.go:356] "Generic (PLEG): container finished" podID="a4af4b40-a245-4146-b652-dd9333b0a4eb" containerID="b6ceba5c65debeece812131e20ac5b8980bcf985cc625dac875dbba96c04d6c4" exitCode=0
Nov 03 12:31:52.843470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:52.843420    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z" event={"ID":"a4af4b40-a245-4146-b652-dd9333b0a4eb","Type":"ContainerDied","Data":"b6ceba5c65debeece812131e20ac5b8980bcf985cc625dac875dbba96c04d6c4"}
Nov 03 12:31:53.989760 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:53.989730    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:54.028054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.028032    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca\") pod \"a4af4b40-a245-4146-b652-dd9333b0a4eb\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") "
Nov 03 12:31:54.028200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.028085    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr\" (UniqueName: \"kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr\") pod \"a4af4b40-a245-4146-b652-dd9333b0a4eb\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") "
Nov 03 12:31:54.028265 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.028225    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca\") pod \"a4af4b40-a245-4146-b652-dd9333b0a4eb\" (UID: \"a4af4b40-a245-4146-b652-dd9333b0a4eb\") "
Nov 03 12:31:54.028405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.028382    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca" pod "a4af4b40-a245-4146-b652-dd9333b0a4eb" (UID: "a4af4b40-a245-4146-b652-dd9333b0a4eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:54.028673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.028653    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:54.030439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.030413    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca" pod "a4af4b40-a245-4146-b652-dd9333b0a4eb" (UID: "a4af4b40-a245-4146-b652-dd9333b0a4eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:54.030685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.030665    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr" pod "a4af4b40-a245-4146-b652-dd9333b0a4eb" (UID: "a4af4b40-a245-4146-b652-dd9333b0a4eb"). InnerVolumeSpecName "kube-api-access-2p9jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:54.129924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.129838    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4af4b40-a245-4146-b652-dd9333b0a4eb-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:54.129924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.129867    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p9jr\" (UniqueName: \"kubernetes.io/projected/a4af4b40-a245-4146-b652-dd9333b0a4eb-kube-api-access-2p9jr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:54.857328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.857233    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z" event={"ID":"a4af4b40-a245-4146-b652-dd9333b0a4eb","Type":"ContainerDied","Data":"1b6f0b5fe4e409a1d0ad06b62d470b6a33e98da6be9d0e253f5e777f324963d2"}
Nov 03 12:31:54.857328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.857265    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"
Nov 03 12:31:54.857328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:54.857290    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b6f0b5fe4e409a1d0ad06b62d470b6a33e98da6be9d0e253f5e777f324963d2"
Nov 03 12:31:56.258074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.258048    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:31:56.367584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367552    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.367777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367596    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.367777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367644    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.367777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367668    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.367777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367706    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.368021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367821    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.368021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367883    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.368021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367875    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:56.368021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367948    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:56.368021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.367992    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.368354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368030    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo\") pod \"b091315f-b67b-450d-8c6a-191c8e388320\" (UID: \"b091315f-b67b-450d-8c6a-191c8e388320\") "
Nov 03 12:31:56.368354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368140    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:56.368354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368261    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:56.368687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368361    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:56.368687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368621    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.368687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368636    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.368687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368647    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.368687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368663    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091315f-b67b-450d-8c6a-191c8e388320-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.368687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.368681    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.370231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.370202    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:31:56.370353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.370237    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "kube-api-access-rcxw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:56.370416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.370372    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:31:56.370687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.370668    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token" pod "b091315f-b67b-450d-8c6a-191c8e388320" (UID: "b091315f-b67b-450d-8c6a-191c8e388320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:56.490712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.490613    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.490712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.490656    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091315f-b67b-450d-8c6a-191c8e388320-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.490712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.490671    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxw8\" (UniqueName: \"kubernetes.io/projected/b091315f-b67b-450d-8c6a-191c8e388320-kube-api-access-rcxw8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.490712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.490688    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091315f-b67b-450d-8c6a-191c8e388320-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:56.876443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.876349    4546 generic.go:356] "Generic (PLEG): container finished" podID="b091315f-b67b-450d-8c6a-191c8e388320" containerID="7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b" exitCode=0
Nov 03 12:31:56.876443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.876429    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"
Nov 03 12:31:56.876443 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.876431    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" event={"ID":"b091315f-b67b-450d-8c6a-191c8e388320","Type":"ContainerDied","Data":"7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b"}
Nov 03 12:31:56.876779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.876499    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp" event={"ID":"b091315f-b67b-450d-8c6a-191c8e388320","Type":"ContainerDied","Data":"19fa16ef9a7b943c7f997d2a02b152ac67b9c386378d37e80658b6f5f9c21331"}
Nov 03 12:31:56.876779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.876533    4546 scope.go:122] "RemoveContainer" containerID="7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b"
Nov 03 12:31:56.889894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.889875    4546 scope.go:122] "RemoveContainer" containerID="bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"
Nov 03 12:31:56.902659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.902618    4546 scope.go:122] "RemoveContainer" containerID="7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b"
Nov 03 12:31:56.903000 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:56.902974    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b\": container with ID starting with 7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b not found: ID does not exist" containerID="7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b"
Nov 03 12:31:56.903128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.903018    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b"} err="failed to get container status \"7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b\": rpc error: code = NotFound desc = could not find container \"7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b\": container with ID starting with 7775df4e4e63de69a812c034220547a0e5e49c46e4d920f2496bbf848ed2b87b not found: ID does not exist"
Nov 03 12:31:56.903128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.903038    4546 scope.go:122] "RemoveContainer" containerID="bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"
Nov 03 12:31:56.903336 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:31:56.903317    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889\": container with ID starting with bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889 not found: ID does not exist" containerID="bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"
Nov 03 12:31:56.903406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:56.903337    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889"} err="failed to get container status \"bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889\": rpc error: code = NotFound desc = could not find container \"bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889\": container with ID starting with bfc0d931d76e057370190cb7ec0601f6ef20c881b6d942efabba2307771c4889 not found: ID does not exist"
Nov 03 12:31:57.045377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:57.045338    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"]
Nov 03 12:31:57.085893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:57.084950    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c02ae4-openshift-default-78f85c565c-glvsp"]
Nov 03 12:31:58.573871 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.573838    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b091315f-b67b-450d-8c6a-191c8e388320" path="/var/lib/kubelet/pods/b091315f-b67b-450d-8c6a-191c8e388320/volumes"
Nov 03 12:31:58.797884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.797838    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"]
Nov 03 12:31:58.798480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798459    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a4af4b40-a245-4146-b652-dd9333b0a4eb" containerName="revoke-keys"
Nov 03 12:31:58.798480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798481    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4af4b40-a245-4146-b652-dd9333b0a4eb" containerName="revoke-keys"
Nov 03 12:31:58.798674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798516    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy"
Nov 03 12:31:58.798674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798523    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy"
Nov 03 12:31:58.798674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798532    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy"
Nov 03 12:31:58.798674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798540    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy"
Nov 03 12:31:58.798922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798678    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4af4b40-a245-4146-b652-dd9333b0a4eb" containerName="revoke-keys"
Nov 03 12:31:58.798922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798696    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy"
Nov 03 12:31:58.798922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.798704    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b091315f-b67b-450d-8c6a-191c8e388320" containerName="istio-proxy"
Nov 03 12:31:58.803500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.803478    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:58.855398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.855329    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"]
Nov 03 12:31:58.897113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.897086    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:31:58.897113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.897122    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:31:58.897326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.897128    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:31:58.897676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.897662    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:31:58.956325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.956292    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:58.956506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.956419    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:58.956506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:58.956503    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfzl\" (UniqueName: \"kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.057509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.057473    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.057724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.057564    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfzl\" (UniqueName: \"kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.057724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.057672    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.058303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.058283    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.059968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.059951    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.110379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.110304    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfzl\" (UniqueName: \"kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl\") pod \"maas-api-revoke-sub-33bf2953-lgsgm\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.117192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.117172    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:31:59.347226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.347193    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"]
Nov 03 12:31:59.351661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.351641    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.405255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.405178    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"]
Nov 03 12:31:59.462013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.461977    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rld9h\" (UniqueName: \"kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.462013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.462019    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.462232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.462145    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.506397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.506364    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"]
Nov 03 12:31:59.507037 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:31:59.507010    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21358f3_7294_4c0c_a074_1175b9cca481.slice/crio-20607f8cbaa5fbbaa308cfb133d9db9411eb163f00206ef401fff63f90575285 WatchSource:0}: Error finding container 20607f8cbaa5fbbaa308cfb133d9db9411eb163f00206ef401fff63f90575285: Status 404 returned error can't find the container with id 20607f8cbaa5fbbaa308cfb133d9db9411eb163f00206ef401fff63f90575285
Nov 03 12:31:59.563233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.563199    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.563391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.563362    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rld9h\" (UniqueName: \"kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.563462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.563402    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.563824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.563805    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.565913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.565887    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.616390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.616354    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rld9h\" (UniqueName: \"kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h\") pod \"maas-api-revoke-sub-bad5d84d-8ghrl\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.665943 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.665871    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:31:59.901225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.901190    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm" event={"ID":"a21358f3-7294-4c0c-a074-1175b9cca481","Type":"ContainerStarted","Data":"e9647365c7734853705b3f3c218783447f4e7cd44d79d4e41574c636258e746e"}
Nov 03 12:31:59.901418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:31:59.901246    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm" event={"ID":"a21358f3-7294-4c0c-a074-1175b9cca481","Type":"ContainerStarted","Data":"20607f8cbaa5fbbaa308cfb133d9db9411eb163f00206ef401fff63f90575285"}
Nov 03 12:32:00.010043 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.009967    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm" podStartSLOduration=2.009946251 podStartE2EDuration="2.009946251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:00.005192607 +0000 UTC m=+2461.576776540" watchObservedRunningTime="2026-11-03 12:32:00.009946251 +0000 UTC m=+2461.581530164"
Nov 03 12:32:00.037850 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:00.037815    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd860e474_de0b_4aa8_97af_34315272aaa2.slice/crio-804dff288abd84a32692184cabe28aa26b63743b366727553d1d3ebba4d96fc6 WatchSource:0}: Error finding container 804dff288abd84a32692184cabe28aa26b63743b366727553d1d3ebba4d96fc6: Status 404 returned error can't find the container with id 804dff288abd84a32692184cabe28aa26b63743b366727553d1d3ebba4d96fc6
Nov 03 12:32:00.054535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.054513    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"]
Nov 03 12:32:00.678895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.678810    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"]
Nov 03 12:32:00.684362 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.684337    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.725069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.725034    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"]
Nov 03 12:32:00.728977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.728959    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.767896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.767861    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bb1b51-openshift-default-dockercfg-b9g7k\""
Nov 03 12:32:00.771929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.771897    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"]
Nov 03 12:32:00.775338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775314    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775348    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775384    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd65t\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775520    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775618    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775767    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775821    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775913 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775905    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.775992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.775927    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.784075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.784046    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"]
Nov 03 12:32:00.877244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877210    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877269    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.877379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877320    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877474    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4p2h\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.877552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877515    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877556    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877698    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.877802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877725    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.877862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877807    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877819    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877892    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.877979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877916    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.877979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877939    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.877979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.877960    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.878179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878006    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.878179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878096    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.878364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878177    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.878364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878222    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.878364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878262    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.878364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878269    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.878364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878293    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.878734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878404    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd65t\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.878920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.878901    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.879920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.879898    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.880893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.880874    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.909510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.909477    4546 generic.go:356] "Generic (PLEG): container finished" podID="d860e474-de0b-4aa8-97af-34315272aaa2" containerID="a0ac8368da9017b310f4f0f657fbed47ae18656c43c92d44e251a70920959a1f" exitCode=0
Nov 03 12:32:00.909671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.909555    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl" event={"ID":"d860e474-de0b-4aa8-97af-34315272aaa2","Type":"ContainerDied","Data":"a0ac8368da9017b310f4f0f657fbed47ae18656c43c92d44e251a70920959a1f"}
Nov 03 12:32:00.909671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.909608    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl" event={"ID":"d860e474-de0b-4aa8-97af-34315272aaa2","Type":"ContainerStarted","Data":"804dff288abd84a32692184cabe28aa26b63743b366727553d1d3ebba4d96fc6"}
Nov 03 12:32:00.911263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.911239    4546 generic.go:356] "Generic (PLEG): container finished" podID="a21358f3-7294-4c0c-a074-1175b9cca481" containerID="e9647365c7734853705b3f3c218783447f4e7cd44d79d4e41574c636258e746e" exitCode=0
Nov 03 12:32:00.911377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.911316    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm" event={"ID":"a21358f3-7294-4c0c-a074-1175b9cca481","Type":"ContainerDied","Data":"e9647365c7734853705b3f3c218783447f4e7cd44d79d4e41574c636258e746e"}
Nov 03 12:32:00.931264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.931179    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.931931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.931910    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd65t\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:00.979254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979221    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p2h\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979268    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979285    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979326    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979356    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979383    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979420    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979455    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979530    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979865    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.979908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979872    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.980090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.979932    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.980090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.980046    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.980380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.980359    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.981887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.981867    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.982009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.981991    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:00.998103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:00.998080    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:01.033279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.033220    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:01.033473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.033359    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p2h\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:01.052488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.052450    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:01.414617 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:01.414588    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb010bc1e_4478_431a_adb7_fb4f5e9ba9a4.slice/crio-f1478dd78bd46e05e66221c1164f6cb26acee22a902697df75d7a491644cb28f WatchSource:0}: Error finding container f1478dd78bd46e05e66221c1164f6cb26acee22a902697df75d7a491644cb28f: Status 404 returned error can't find the container with id f1478dd78bd46e05e66221c1164f6cb26acee22a902697df75d7a491644cb28f
Nov 03 12:32:01.415343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.415318    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"]
Nov 03 12:32:01.417305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.417275    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:01.417398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.417337    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:01.417398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.417366    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:01.466053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.466023    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"]
Nov 03 12:32:01.475017 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:01.474985    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0e2b2_b5ad_4826_9753_a8da70d28a10.slice/crio-dde0be3d13da91b12c72259fe3855f01e2dde680a42aaf837bf4c8c478794c56 WatchSource:0}: Error finding container dde0be3d13da91b12c72259fe3855f01e2dde680a42aaf837bf4c8c478794c56: Status 404 returned error can't find the container with id dde0be3d13da91b12c72259fe3855f01e2dde680a42aaf837bf4c8c478794c56
Nov 03 12:32:01.478007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.477965    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:01.478109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.478056    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:01.478109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.478100    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:01.920934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.920892    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" event={"ID":"cbc0e2b2-b5ad-4826-9753-a8da70d28a10","Type":"ContainerStarted","Data":"55aef8c02249789c450391f6ae3781d26358c64ec6dd465e77ff479f60718bc7"}
Nov 03 12:32:01.921407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.920965    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" event={"ID":"cbc0e2b2-b5ad-4826-9753-a8da70d28a10","Type":"ContainerStarted","Data":"dde0be3d13da91b12c72259fe3855f01e2dde680a42aaf837bf4c8c478794c56"}
Nov 03 12:32:01.922808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.922778    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" event={"ID":"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4","Type":"ContainerStarted","Data":"43746d6d7530aa9c3deb2b9a1176d8d796785374efa48119774743cb4f6934ef"}
Nov 03 12:32:01.922958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.922827    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" event={"ID":"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4","Type":"ContainerStarted","Data":"f1478dd78bd46e05e66221c1164f6cb26acee22a902697df75d7a491644cb28f"}
Nov 03 12:32:01.998876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:01.998849    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:02.034769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.034374    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podStartSLOduration=2.034355776 podStartE2EDuration="2.034355776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:02.030093831 +0000 UTC m=+2463.601677764" watchObservedRunningTime="2026-11-03 12:32:02.034355776 +0000 UTC m=+2463.605939692"
Nov 03 12:32:02.053422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.053396    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:02.054816 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.054792    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:02.054949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.054837    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:02.072704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.072673    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:32:02.134640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.131929    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:32:02.176414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.173540    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podStartSLOduration=2.173519406 podStartE2EDuration="2.173519406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:02.164964556 +0000 UTC m=+2463.736548468" watchObservedRunningTime="2026-11-03 12:32:02.173519406 +0000 UTC m=+2463.745103317"
Nov 03 12:32:02.196119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.196088    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca\") pod \"d860e474-de0b-4aa8-97af-34315272aaa2\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") "
Nov 03 12:32:02.196305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.196129    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca\") pod \"d860e474-de0b-4aa8-97af-34315272aaa2\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") "
Nov 03 12:32:02.196305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.196190    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h\" (UniqueName: \"kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h\") pod \"d860e474-de0b-4aa8-97af-34315272aaa2\" (UID: \"d860e474-de0b-4aa8-97af-34315272aaa2\") "
Nov 03 12:32:02.196588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.196560    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca" pod "d860e474-de0b-4aa8-97af-34315272aaa2" (UID: "d860e474-de0b-4aa8-97af-34315272aaa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:02.196837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.196817    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:02.198438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.198414    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca" pod "d860e474-de0b-4aa8-97af-34315272aaa2" (UID: "d860e474-de0b-4aa8-97af-34315272aaa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:02.198516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.198495    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h" pod "d860e474-de0b-4aa8-97af-34315272aaa2" (UID: "d860e474-de0b-4aa8-97af-34315272aaa2"). InnerVolumeSpecName "kube-api-access-rld9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:02.298778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.298213    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca\") pod \"a21358f3-7294-4c0c-a074-1175b9cca481\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") "
Nov 03 12:32:02.298778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.298331    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl\" (UniqueName: \"kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl\") pod \"a21358f3-7294-4c0c-a074-1175b9cca481\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") "
Nov 03 12:32:02.298778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.298515    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca\") pod \"a21358f3-7294-4c0c-a074-1175b9cca481\" (UID: \"a21358f3-7294-4c0c-a074-1175b9cca481\") "
Nov 03 12:32:02.299087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.298969    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860e474-de0b-4aa8-97af-34315272aaa2-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:02.299087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.298992    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rld9h\" (UniqueName: \"kubernetes.io/projected/d860e474-de0b-4aa8-97af-34315272aaa2-kube-api-access-rld9h\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:02.299742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.299328    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca" pod "a21358f3-7294-4c0c-a074-1175b9cca481" (UID: "a21358f3-7294-4c0c-a074-1175b9cca481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:02.301508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.301481    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca" pod "a21358f3-7294-4c0c-a074-1175b9cca481" (UID: "a21358f3-7294-4c0c-a074-1175b9cca481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:02.302127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.302103    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl" pod "a21358f3-7294-4c0c-a074-1175b9cca481" (UID: "a21358f3-7294-4c0c-a074-1175b9cca481"). InnerVolumeSpecName "kube-api-access-nmfzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:02.399719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.399686    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfzl\" (UniqueName: \"kubernetes.io/projected/a21358f3-7294-4c0c-a074-1175b9cca481-kube-api-access-nmfzl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:02.399719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.399715    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:02.399719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.399728    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21358f3-7294-4c0c-a074-1175b9cca481-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:02.973473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.973442    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"
Nov 03 12:32:02.973473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.973451    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl" event={"ID":"d860e474-de0b-4aa8-97af-34315272aaa2","Type":"ContainerDied","Data":"804dff288abd84a32692184cabe28aa26b63743b366727553d1d3ebba4d96fc6"}
Nov 03 12:32:02.973908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.973531    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="804dff288abd84a32692184cabe28aa26b63743b366727553d1d3ebba4d96fc6"
Nov 03 12:32:02.974851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.974828    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"
Nov 03 12:32:02.975012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.974853    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm" event={"ID":"a21358f3-7294-4c0c-a074-1175b9cca481","Type":"ContainerDied","Data":"20607f8cbaa5fbbaa308cfb133d9db9411eb163f00206ef401fff63f90575285"}
Nov 03 12:32:02.975012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:02.974895    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20607f8cbaa5fbbaa308cfb133d9db9411eb163f00206ef401fff63f90575285"
Nov 03 12:32:03.000071 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:03.000043    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 03 12:32:03.000204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:03.000094    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 03 12:32:03.059040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:03.058970    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:03.059267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:03.059051    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:03.999361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:03.999329    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 03 12:32:03.999907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:03.999380    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 03 12:32:04.085564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.076965    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:04.085564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.077029    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:04.640241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.640205    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"]
Nov 03 12:32:04.659568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.659524    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a21358f3-7294-4c0c-a074-1175b9cca481" containerName="revoke-keys"
Nov 03 12:32:04.659568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.659557    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21358f3-7294-4c0c-a074-1175b9cca481" containerName="revoke-keys"
Nov 03 12:32:04.659861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.659597    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d860e474-de0b-4aa8-97af-34315272aaa2" containerName="revoke-keys"
Nov 03 12:32:04.659861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.659605    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d860e474-de0b-4aa8-97af-34315272aaa2" containerName="revoke-keys"
Nov 03 12:32:04.659861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.659724    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21358f3-7294-4c0c-a074-1175b9cca481" containerName="revoke-keys"
Nov 03 12:32:04.659861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.659762    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d860e474-de0b-4aa8-97af-34315272aaa2" containerName="revoke-keys"
Nov 03 12:32:04.664987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.664968    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.693517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.693485    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"]
Nov 03 12:32:04.698823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.698797    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.698935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.698899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszpb\" (UniqueName: \"kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb\") pod \"payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.763223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.763191    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2daac7\""
Nov 03 12:32:04.763798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.763774    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2daac7-dockercfg-x942c\""
Nov 03 12:32:04.799674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.799641    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszpb\" (UniqueName: \"kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb\") pod \"payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.799884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.799869    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.800598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.800571    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.860631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.860506    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszpb\" (UniqueName: \"kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb\") pod \"payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.973855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.973810    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"]
Nov 03 12:32:04.978010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.977987    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:04.979244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.979224    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:04.999259 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.999235    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 03 12:32:04.999374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:04.999284    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 03 12:32:05.002857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.001967    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.002857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.001994    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8dpv\" (UniqueName: \"kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.002857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.002079    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.072699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.061224    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:32:05.072699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.066571    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:32:05.072699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.066626    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:32:05.072699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.066683    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:32:05.072699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.067956    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:05.072699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.067999    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:05.076546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.075969    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"]
Nov 03 12:32:05.099403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.095524    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"]
Nov 03 12:32:05.100883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.100862    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.104185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.103108    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.104185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.103233    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.104185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.103248    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8dpv\" (UniqueName: \"kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.104185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.104139    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.106207 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.106186    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.177263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.177205    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8dpv\" (UniqueName: \"kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv\") pod \"maas-api-revoke-sub-013ab3ec-zzqjk\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.195618 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.195525    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"]
Nov 03 12:32:05.204405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.204372    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.204511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.204458    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62wxv\" (UniqueName: \"kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.297811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.297712    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:05.305460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.305427    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.305609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.305487    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62wxv\" (UniqueName: \"kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.306283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.306253    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.362431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.362408    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62wxv\" (UniqueName: \"kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv\") pod \"payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.500479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.500434    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"]
Nov 03 12:32:05.501928 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:05.501898    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14a0f84_2f47_41a9_9b7b_60406dbe48e7.slice/crio-174e71b2b94311d38133516c3476f05964311d1a180a58d372a9d69ccaf1241d WatchSource:0}: Error finding container 174e71b2b94311d38133516c3476f05964311d1a180a58d372a9d69ccaf1241d: Status 404 returned error can't find the container with id 174e71b2b94311d38133516c3476f05964311d1a180a58d372a9d69ccaf1241d
Nov 03 12:32:05.505199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.503991    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:05.771090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.771053    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"]
Nov 03 12:32:05.772991 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:05.772956    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6069fd59_2d21_4fcb_bf1a_deb14a1526d3.slice/crio-d07c2e05ffaa2640b2a033e05625a3740c091df4db39cf26266b863d94e958f1 WatchSource:0}: Error finding container d07c2e05ffaa2640b2a033e05625a3740c091df4db39cf26266b863d94e958f1: Status 404 returned error can't find the container with id d07c2e05ffaa2640b2a033e05625a3740c091df4db39cf26266b863d94e958f1
Nov 03 12:32:05.996624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:05.996587    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"]
Nov 03 12:32:06.001559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.001528    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 03 12:32:06.001689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.001599    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 03 12:32:06.041979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.041941    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk" event={"ID":"6069fd59-2d21-4fcb-bf1a-deb14a1526d3","Type":"ContainerStarted","Data":"d07c2e05ffaa2640b2a033e05625a3740c091df4db39cf26266b863d94e958f1"}
Nov 03 12:32:06.044479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.044447    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd" event={"ID":"f14a0f84-2f47-41a9-9b7b-60406dbe48e7","Type":"ContainerStarted","Data":"4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54"}
Nov 03 12:32:06.044688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.044670    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd" event={"ID":"f14a0f84-2f47-41a9-9b7b-60406dbe48e7","Type":"ContainerStarted","Data":"174e71b2b94311d38133516c3476f05964311d1a180a58d372a9d69ccaf1241d"}
Nov 03 12:32:06.045860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.045842    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:06.056049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.056022    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:06.056243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.056221    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:06.071616 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:06.071579    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaefc472_3976_4c96_8bcc_95ac49485bdd.slice/crio-2f88dfd1b98d3f945ac369244e94dc864996b4e8ea950ea0fe1208369457892a WatchSource:0}: Error finding container 2f88dfd1b98d3f945ac369244e94dc864996b4e8ea950ea0fe1208369457892a: Status 404 returned error can't find the container with id 2f88dfd1b98d3f945ac369244e94dc864996b4e8ea950ea0fe1208369457892a
Nov 03 12:32:06.999129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.999082    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 03 12:32:06.999364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:06.999146    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 03 12:32:07.056773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.053076    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:07.056773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.053134    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:07.081405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.081365    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk" event={"ID":"6069fd59-2d21-4fcb-bf1a-deb14a1526d3","Type":"ContainerStarted","Data":"670656fc8f6827c9063c7f09d10b3bb1d42d52eb2c486ac9f7da306d510b4ae5"}
Nov 03 12:32:07.084100 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.084068    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" event={"ID":"baefc472-3976-4c96-8bcc-95ac49485bdd","Type":"ContainerStarted","Data":"9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3"}
Nov 03 12:32:07.084252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.084114    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" event={"ID":"baefc472-3976-4c96-8bcc-95ac49485bdd","Type":"ContainerStarted","Data":"2f88dfd1b98d3f945ac369244e94dc864996b4e8ea950ea0fe1208369457892a"}
Nov 03 12:32:07.084252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.084142    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:07.194852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.192825    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk" podStartSLOduration=3.192810644 podStartE2EDuration="3.192810644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:07.192659768 +0000 UTC m=+2468.764243680" watchObservedRunningTime="2026-11-03 12:32:07.192810644 +0000 UTC m=+2468.764394555"
Nov 03 12:32:07.194852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.193902    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd" podStartSLOduration=3.193891882 podStartE2EDuration="3.193891882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:06.155206447 +0000 UTC m=+2467.726790359" watchObservedRunningTime="2026-11-03 12:32:07.193891882 +0000 UTC m=+2468.765475793"
Nov 03 12:32:07.300872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.300035    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" podStartSLOduration=2.300019025 podStartE2EDuration="2.300019025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:07.298961365 +0000 UTC m=+2468.870545277" watchObservedRunningTime="2026-11-03 12:32:07.300019025 +0000 UTC m=+2468.871602936"
Nov 03 12:32:07.688388 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:32:07.688279    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:32:07.999233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.999191    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 03 12:32:07.999402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:07.999259    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 03 12:32:08.054101 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:08.054064    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 12:32:08.054300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:08.054127    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 12:32:08.092484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:08.092454    4546 generic.go:356] "Generic (PLEG): container finished" podID="6069fd59-2d21-4fcb-bf1a-deb14a1526d3" containerID="670656fc8f6827c9063c7f09d10b3bb1d42d52eb2c486ac9f7da306d510b4ae5" exitCode=0
Nov 03 12:32:08.093067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:08.093034    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk" event={"ID":"6069fd59-2d21-4fcb-bf1a-deb14a1526d3","Type":"ContainerDied","Data":"670656fc8f6827c9063c7f09d10b3bb1d42d52eb2c486ac9f7da306d510b4ae5"}
Nov 03 12:32:09.081739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.081703    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:09.082438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.082422    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:09.083496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.083476    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:32:09.092297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.092276    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:09.092668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.092643    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:09.095645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.095623    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:32:09.305408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.305384    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:09.369270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.369180    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca\") pod \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") "
Nov 03 12:32:09.369429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.369410    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv\" (UniqueName: \"kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv\") pod \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") "
Nov 03 12:32:09.369483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.369451    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca\") pod \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\" (UID: \"6069fd59-2d21-4fcb-bf1a-deb14a1526d3\") "
Nov 03 12:32:09.369908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.369616    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca" pod "6069fd59-2d21-4fcb-bf1a-deb14a1526d3" (UID: "6069fd59-2d21-4fcb-bf1a-deb14a1526d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:09.370073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.370050    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:09.375799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.372059    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca" pod "6069fd59-2d21-4fcb-bf1a-deb14a1526d3" (UID: "6069fd59-2d21-4fcb-bf1a-deb14a1526d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:09.375799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.372263    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv" pod "6069fd59-2d21-4fcb-bf1a-deb14a1526d3" (UID: "6069fd59-2d21-4fcb-bf1a-deb14a1526d3"). InnerVolumeSpecName "kube-api-access-g8dpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:09.471526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.471490    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8dpv\" (UniqueName: \"kubernetes.io/projected/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-kube-api-access-g8dpv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:09.471688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:09.471530    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6069fd59-2d21-4fcb-bf1a-deb14a1526d3-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:10.108045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:10.108016    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"
Nov 03 12:32:10.108498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:10.108052    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk" event={"ID":"6069fd59-2d21-4fcb-bf1a-deb14a1526d3","Type":"ContainerDied","Data":"d07c2e05ffaa2640b2a033e05625a3740c091df4db39cf26266b863d94e958f1"}
Nov 03 12:32:10.108498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:10.108097    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d07c2e05ffaa2640b2a033e05625a3740c091df4db39cf26266b863d94e958f1"
Nov 03 12:32:15.311942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.311910    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"]
Nov 03 12:32:15.312656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.312633    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6069fd59-2d21-4fcb-bf1a-deb14a1526d3" containerName="revoke-keys"
Nov 03 12:32:15.312786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.312658    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6069fd59-2d21-4fcb-bf1a-deb14a1526d3" containerName="revoke-keys"
Nov 03 12:32:15.312900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.312884    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6069fd59-2d21-4fcb-bf1a-deb14a1526d3" containerName="revoke-keys"
Nov 03 12:32:15.316713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.316693    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.411383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.411357    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:32:15.411383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.411353    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:32:15.411600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.411402    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:32:15.411600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.411452    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:32:15.415276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.415255    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"]
Nov 03 12:32:15.444537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.444511    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.444695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.444588    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkj87\" (UniqueName: \"kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.444824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.444805    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.546255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.546227    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.546446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.546315    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.546446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.546385    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkj87\" (UniqueName: \"kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.546930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.546903    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.549444 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.549405    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.602962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.602887    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkj87\" (UniqueName: \"kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87\") pod \"maas-api-revoke-sub-c32bc120-fb4ps\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.630423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.630399    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:15.693856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.693826    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"]
Nov 03 12:32:15.706239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.706204    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.748958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.748927    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.749361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.748966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lmj\" (UniqueName: \"kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.749361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.749037    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.753466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.753442    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"]
Nov 03 12:32:15.850315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.850282    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.850315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.850313    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lmj\" (UniqueName: \"kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.850594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.850336    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.851021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.850996    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.852775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.852741    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:15.903710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:15.903657    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lmj\" (UniqueName: \"kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj\") pod \"maas-api-revoke-sub-616c7952-tjngl\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:16.002640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:16.002609    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"]
Nov 03 12:32:16.003139 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:16.003117    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f1d44e_b23f_430b_a484_bba1cda31a06.slice/crio-ef44cff1ffc27e2ba17ea5f2eb1a3c837fca493de5918aa1e829ff22634f47e8 WatchSource:0}: Error finding container ef44cff1ffc27e2ba17ea5f2eb1a3c837fca493de5918aa1e829ff22634f47e8: Status 404 returned error can't find the container with id ef44cff1ffc27e2ba17ea5f2eb1a3c837fca493de5918aa1e829ff22634f47e8
Nov 03 12:32:16.020655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:16.020635    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:16.150283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:16.150250    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps" event={"ID":"b3f1d44e-b23f-430b-a484-bba1cda31a06","Type":"ContainerStarted","Data":"ef44cff1ffc27e2ba17ea5f2eb1a3c837fca493de5918aa1e829ff22634f47e8"}
Nov 03 12:32:16.392777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:16.392731    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"]
Nov 03 12:32:16.393205 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:16.393071    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c89775_6b94_4786_8e96_da6bcb0a3530.slice/crio-8e090d090ed80ee2db371561b2e16c6a5962f917f06eba809c61ed63f843821b WatchSource:0}: Error finding container 8e090d090ed80ee2db371561b2e16c6a5962f917f06eba809c61ed63f843821b: Status 404 returned error can't find the container with id 8e090d090ed80ee2db371561b2e16c6a5962f917f06eba809c61ed63f843821b
Nov 03 12:32:17.159430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:17.159387    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl" event={"ID":"24c89775-6b94-4786-8e96-da6bcb0a3530","Type":"ContainerStarted","Data":"98cd3713aad3b49b5be59df1b27cda9036051b59f7f07d8ea6bf595307e5f14a"}
Nov 03 12:32:17.159615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:17.159452    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl" event={"ID":"24c89775-6b94-4786-8e96-da6bcb0a3530","Type":"ContainerStarted","Data":"8e090d090ed80ee2db371561b2e16c6a5962f917f06eba809c61ed63f843821b"}
Nov 03 12:32:17.161192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:17.161158    4546 generic.go:356] "Generic (PLEG): container finished" podID="b3f1d44e-b23f-430b-a484-bba1cda31a06" containerID="7c73ac070eb6bbf36662e796c13612256fb06035ba152778adc49c6957af812b" exitCode=0
Nov 03 12:32:17.161298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:17.161206    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps" event={"ID":"b3f1d44e-b23f-430b-a484-bba1cda31a06","Type":"ContainerDied","Data":"7c73ac070eb6bbf36662e796c13612256fb06035ba152778adc49c6957af812b"}
Nov 03 12:32:17.278247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:17.278188    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl" podStartSLOduration=2.278168102 podStartE2EDuration="2.278168102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:17.26429752 +0000 UTC m=+2478.835881437" watchObservedRunningTime="2026-11-03 12:32:17.278168102 +0000 UTC m=+2478.849752016"
Nov 03 12:32:18.095405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.095374    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:18.171180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.171150    4546 generic.go:356] "Generic (PLEG): container finished" podID="24c89775-6b94-4786-8e96-da6bcb0a3530" containerID="98cd3713aad3b49b5be59df1b27cda9036051b59f7f07d8ea6bf595307e5f14a" exitCode=0
Nov 03 12:32:18.171357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.171231    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl" event={"ID":"24c89775-6b94-4786-8e96-da6bcb0a3530","Type":"ContainerDied","Data":"98cd3713aad3b49b5be59df1b27cda9036051b59f7f07d8ea6bf595307e5f14a"}
Nov 03 12:32:18.351925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.351854    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:18.492220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.492185    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca\") pod \"b3f1d44e-b23f-430b-a484-bba1cda31a06\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") "
Nov 03 12:32:18.492401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.492358    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87\" (UniqueName: \"kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87\") pod \"b3f1d44e-b23f-430b-a484-bba1cda31a06\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") "
Nov 03 12:32:18.492516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.492501    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca\") pod \"b3f1d44e-b23f-430b-a484-bba1cda31a06\" (UID: \"b3f1d44e-b23f-430b-a484-bba1cda31a06\") "
Nov 03 12:32:18.493102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.493022    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca" pod "b3f1d44e-b23f-430b-a484-bba1cda31a06" (UID: "b3f1d44e-b23f-430b-a484-bba1cda31a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:18.494500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.494468    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca" pod "b3f1d44e-b23f-430b-a484-bba1cda31a06" (UID: "b3f1d44e-b23f-430b-a484-bba1cda31a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:18.494700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.494679    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87" pod "b3f1d44e-b23f-430b-a484-bba1cda31a06" (UID: "b3f1d44e-b23f-430b-a484-bba1cda31a06"). InnerVolumeSpecName "kube-api-access-tkj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:18.593735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.593705    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkj87\" (UniqueName: \"kubernetes.io/projected/b3f1d44e-b23f-430b-a484-bba1cda31a06-kube-api-access-tkj87\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:18.593735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.593731    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:18.593735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:18.593759    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f1d44e-b23f-430b-a484-bba1cda31a06-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:19.102422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.102391    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:19.179155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.179104    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"
Nov 03 12:32:19.179366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.179146    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps" event={"ID":"b3f1d44e-b23f-430b-a484-bba1cda31a06","Type":"ContainerDied","Data":"ef44cff1ffc27e2ba17ea5f2eb1a3c837fca493de5918aa1e829ff22634f47e8"}
Nov 03 12:32:19.179366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.179208    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef44cff1ffc27e2ba17ea5f2eb1a3c837fca493de5918aa1e829ff22634f47e8"
Nov 03 12:32:19.348689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.348664    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:19.403514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.403427    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj\" (UniqueName: \"kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj\") pod \"24c89775-6b94-4786-8e96-da6bcb0a3530\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") "
Nov 03 12:32:19.403703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.403597    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca\") pod \"24c89775-6b94-4786-8e96-da6bcb0a3530\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") "
Nov 03 12:32:19.403703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.403655    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca\") pod \"24c89775-6b94-4786-8e96-da6bcb0a3530\" (UID: \"24c89775-6b94-4786-8e96-da6bcb0a3530\") "
Nov 03 12:32:19.403989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.403967    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca" pod "24c89775-6b94-4786-8e96-da6bcb0a3530" (UID: "24c89775-6b94-4786-8e96-da6bcb0a3530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:19.404111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.404098    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:19.405793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.405765    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca" pod "24c89775-6b94-4786-8e96-da6bcb0a3530" (UID: "24c89775-6b94-4786-8e96-da6bcb0a3530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:19.405892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.405792    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj" pod "24c89775-6b94-4786-8e96-da6bcb0a3530" (UID: "24c89775-6b94-4786-8e96-da6bcb0a3530"). InnerVolumeSpecName "kube-api-access-58lmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:19.504841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.504808    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lmj\" (UniqueName: \"kubernetes.io/projected/24c89775-6b94-4786-8e96-da6bcb0a3530-kube-api-access-58lmj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:19.504841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.504836    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c89775-6b94-4786-8e96-da6bcb0a3530-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:19.953950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.953918    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"]
Nov 03 12:32:19.954452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.954440    4546 cpu_manager.go:414] "containerMap: removing container" podUID="24c89775-6b94-4786-8e96-da6bcb0a3530" containerName="revoke-keys"
Nov 03 12:32:19.954514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.954454    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c89775-6b94-4786-8e96-da6bcb0a3530" containerName="revoke-keys"
Nov 03 12:32:19.954514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.954478    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b3f1d44e-b23f-430b-a484-bba1cda31a06" containerName="revoke-keys"
Nov 03 12:32:19.954514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.954484    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f1d44e-b23f-430b-a484-bba1cda31a06" containerName="revoke-keys"
Nov 03 12:32:19.954613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.954565    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c89775-6b94-4786-8e96-da6bcb0a3530" containerName="revoke-keys"
Nov 03 12:32:19.954613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.954578    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f1d44e-b23f-430b-a484-bba1cda31a06" containerName="revoke-keys"
Nov 03 12:32:19.958986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:19.958964    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.015791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.015665    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"]
Nov 03 12:32:20.055629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.055597    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1250b3-openshift-default-dockercfg-sksxt\""
Nov 03 12:32:20.110529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110482    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110560    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110618    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110647    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110740    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbjn\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110800    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110837    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110864    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.111089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.110896    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.190151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.190105    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"
Nov 03 12:32:20.190411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.190108    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl" event={"ID":"24c89775-6b94-4786-8e96-da6bcb0a3530","Type":"ContainerDied","Data":"8e090d090ed80ee2db371561b2e16c6a5962f917f06eba809c61ed63f843821b"}
Nov 03 12:32:20.190411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.190264    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e090d090ed80ee2db371561b2e16c6a5962f917f06eba809c61ed63f843821b"
Nov 03 12:32:20.212575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212503    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.212575 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212558    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.212740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212597    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.212740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212617    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.212740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212674    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbjn\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.212740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212706    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.212740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212734    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212797    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.212830    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.213019    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.213392    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.213583    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.213628    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.213858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.213835    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.215429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.215408    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.215809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.215791    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.266204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.266133    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.266204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.266133    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbjn\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn\") pod \"e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") " pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.273848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.273171    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:20.721036 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:20.720995    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69594464_d692_4da3_8dd3_d31e710455c8.slice/crio-21deedc0b92283e0d24ab7c1a69edd8d7d0e527f6ce904e0f09b55e7638f4cf4 WatchSource:0}: Error finding container 21deedc0b92283e0d24ab7c1a69edd8d7d0e527f6ce904e0f09b55e7638f4cf4: Status 404 returned error can't find the container with id 21deedc0b92283e0d24ab7c1a69edd8d7d0e527f6ce904e0f09b55e7638f4cf4
Nov 03 12:32:20.721175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.721150    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"]
Nov 03 12:32:20.723673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.723642    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:20.723800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.723702    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:20.723800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:20.723733    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:21.201710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:21.201617    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" event={"ID":"69594464-d692-4da3-8dd3-d31e710455c8","Type":"ContainerStarted","Data":"ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"}
Nov 03 12:32:21.201710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:21.201681    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" event={"ID":"69594464-d692-4da3-8dd3-d31e710455c8","Type":"ContainerStarted","Data":"21deedc0b92283e0d24ab7c1a69edd8d7d0e527f6ce904e0f09b55e7638f4cf4"}
Nov 03 12:32:21.273681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:21.273651    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:21.279632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:21.279607    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:21.312394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:21.312347    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podStartSLOduration=2.312334371 podStartE2EDuration="2.312334371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:21.308633516 +0000 UTC m=+2482.880217431" watchObservedRunningTime="2026-11-03 12:32:21.312334371 +0000 UTC m=+2482.883918282"
Nov 03 12:32:22.208214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:22.208182    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:22.209288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:22.209268    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:32:25.558020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:25.557961    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.610512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:25.610469    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.647797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:25.643883    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded"
Nov 03 12:32:25.756875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:25.756830    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:25.810704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:25.805843    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:30.875548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:30.875510    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"]
Nov 03 12:32:30.876013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:30.875724    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy" containerID="cri-o://e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5" gracePeriod=30
Nov 03 12:32:32.977628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:32.977594    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"]
Nov 03 12:32:32.982863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:32.982841    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.039906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.039876    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4t4n\" (UniqueName: \"kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.040161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.040131    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.040290 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.040209    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.044126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.044069    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"]
Nov 03 12:32:33.079309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.079277    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:32:33.079460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.079323    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:32:33.079460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.079371    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:32:33.079759 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.079278    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:32:33.141698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.141672    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4t4n\" (UniqueName: \"kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.141852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.141715    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.141852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.141775    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.142527 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.142507    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.144288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.144270    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.194264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.194242    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4t4n\" (UniqueName: \"kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n\") pod \"maas-api-revoke-sub-6e41ba69-m7lrh\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.296540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.296469    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:33.667454 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:33.667414    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f90670_2e44_4225_adbd_356d5fa485b7.slice/crio-19020137d784776dbacea2f823c1aaa428ed2c36b8a4841155972959d85c3bf1 WatchSource:0}: Error finding container 19020137d784776dbacea2f823c1aaa428ed2c36b8a4841155972959d85c3bf1: Status 404 returned error can't find the container with id 19020137d784776dbacea2f823c1aaa428ed2c36b8a4841155972959d85c3bf1
Nov 03 12:32:33.667613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:33.667594    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"]
Nov 03 12:32:34.317434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:34.317399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh" event={"ID":"38f90670-2e44-4225-adbd-356d5fa485b7","Type":"ContainerStarted","Data":"0cdbaa9ab8efe9057ac98e7f10fd3a96eb1e8b32d398b93cc40d0a0ba3c58b57"}
Nov 03 12:32:34.317886 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:34.317453    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh" event={"ID":"38f90670-2e44-4225-adbd-356d5fa485b7","Type":"ContainerStarted","Data":"19020137d784776dbacea2f823c1aaa428ed2c36b8a4841155972959d85c3bf1"}
Nov 03 12:32:34.423763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:34.423684    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh" podStartSLOduration=2.423667033 podStartE2EDuration="2.423667033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:34.422219743 +0000 UTC m=+2495.993803654" watchObservedRunningTime="2026-11-03 12:32:34.423667033 +0000 UTC m=+2495.995250946"
Nov 03 12:32:35.326009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:35.325979    4546 generic.go:356] "Generic (PLEG): container finished" podID="38f90670-2e44-4225-adbd-356d5fa485b7" containerID="0cdbaa9ab8efe9057ac98e7f10fd3a96eb1e8b32d398b93cc40d0a0ba3c58b57" exitCode=0
Nov 03 12:32:35.326407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:35.326063    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh" event={"ID":"38f90670-2e44-4225-adbd-356d5fa485b7","Type":"ContainerDied","Data":"0cdbaa9ab8efe9057ac98e7f10fd3a96eb1e8b32d398b93cc40d0a0ba3c58b57"}
Nov 03 12:32:36.157798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.157717    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179545    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179612    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179652    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179715    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179803    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179899    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.179964    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.180022    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.182187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.180126    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k\") pod \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\" (UID: \"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c\") "
Nov 03 12:32:36.183352 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.183010    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:36.184177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.184156    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.184636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.184606    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:36.185109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.185088    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:36.185423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.185402    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:36.186679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.186579    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:32:36.186679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.186585    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:36.228182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.228127    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:32:36.239006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.238972    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "kube-api-access-pbw4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:36.247354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.247283    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token" pod "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" (UID: "6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:36.285612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285582    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285612    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285625    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbw4k\" (UniqueName: \"kubernetes.io/projected/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-kube-api-access-pbw4k\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285640    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285651    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285662    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285673    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.285898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.285689    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.336115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.336086    4546 generic.go:356] "Generic (PLEG): container finished" podID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerID="e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5" exitCode=0
Nov 03 12:32:36.336563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.336152    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"
Nov 03 12:32:36.336563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.336169    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" event={"ID":"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c","Type":"ContainerDied","Data":"e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5"}
Nov 03 12:32:36.336563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.336237    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx" event={"ID":"6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c","Type":"ContainerDied","Data":"65c5628dc28bb66c0b359546ad6039d89a4c59a80006ec004c258b1d9b2bfc6e"}
Nov 03 12:32:36.336563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.336260    4546 scope.go:122] "RemoveContainer" containerID="e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5"
Nov 03 12:32:36.350010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.349984    4546 scope.go:122] "RemoveContainer" containerID="e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5"
Nov 03 12:32:36.350379 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:32:36.350352    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5\": container with ID starting with e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5 not found: ID does not exist" containerID="e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5"
Nov 03 12:32:36.350505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.350392    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5"} err="failed to get container status \"e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5\": rpc error: code = NotFound desc = could not find container \"e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5\": container with ID starting with e475be4dbe10584a90cb01329885c2cab8a48bbd07f6e044751d5606ac242dd5 not found: ID does not exist"
Nov 03 12:32:36.478425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.478400    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:36.490932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.490907    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"]
Nov 03 12:32:36.494596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.494574    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ltzvx"]
Nov 03 12:32:36.573480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.573455    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" path="/var/lib/kubelet/pods/6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c/volumes"
Nov 03 12:32:36.590094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.590073    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n\" (UniqueName: \"kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n\") pod \"38f90670-2e44-4225-adbd-356d5fa485b7\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") "
Nov 03 12:32:36.590211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.590175    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca\") pod \"38f90670-2e44-4225-adbd-356d5fa485b7\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") "
Nov 03 12:32:36.590261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.590239    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca\") pod \"38f90670-2e44-4225-adbd-356d5fa485b7\" (UID: \"38f90670-2e44-4225-adbd-356d5fa485b7\") "
Nov 03 12:32:36.590579 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.590546    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca" pod "38f90670-2e44-4225-adbd-356d5fa485b7" (UID: "38f90670-2e44-4225-adbd-356d5fa485b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:36.592162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.592139    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n" pod "38f90670-2e44-4225-adbd-356d5fa485b7" (UID: "38f90670-2e44-4225-adbd-356d5fa485b7"). InnerVolumeSpecName "kube-api-access-h4t4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:36.592306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.592286    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca" pod "38f90670-2e44-4225-adbd-356d5fa485b7" (UID: "38f90670-2e44-4225-adbd-356d5fa485b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:36.692237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.692158    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.692237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.692190    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f90670-2e44-4225-adbd-356d5fa485b7-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:36.692237 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:36.692207    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4t4n\" (UniqueName: \"kubernetes.io/projected/38f90670-2e44-4225-adbd-356d5fa485b7-kube-api-access-h4t4n\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:37.343426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:37.343396    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"
Nov 03 12:32:37.343880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:37.343432    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh" event={"ID":"38f90670-2e44-4225-adbd-356d5fa485b7","Type":"ContainerDied","Data":"19020137d784776dbacea2f823c1aaa428ed2c36b8a4841155972959d85c3bf1"}
Nov 03 12:32:37.343880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:37.343518    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19020137d784776dbacea2f823c1aaa428ed2c36b8a4841155972959d85c3bf1"
Nov 03 12:32:37.651797 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:32:37.651688    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:32:38.919098 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:38.919067    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"]
Nov 03 12:32:38.919524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:38.919320    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd" podUID="f14a0f84-2f47-41a9-9b7b-60406dbe48e7" containerName="payload-processing" containerID="cri-o://4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54" gracePeriod=30
Nov 03 12:32:38.967331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:38.967305    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"]
Nov 03 12:32:38.967508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:38.967489    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" podUID="baefc472-3976-4c96-8bcc-95ac49485bdd" containerName="payload-pre-processing" containerID="cri-o://9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3" gracePeriod=30
Nov 03 12:32:39.101582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.101540    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" podUID="baefc472-3976-4c96-8bcc-95ac49485bdd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.234:9004: connect: connection refused"
Nov 03 12:32:39.250246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.250215    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:39.275552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.275525    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:39.322340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.322308    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb\" (UniqueName: \"kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb\") pod \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") "
Nov 03 12:32:39.322520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.322436    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume\") pod \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\" (UID: \"f14a0f84-2f47-41a9-9b7b-60406dbe48e7\") "
Nov 03 12:32:39.322588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.322565    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv\" (UniqueName: \"kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv\") pod \"baefc472-3976-4c96-8bcc-95ac49485bdd\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") "
Nov 03 12:32:39.322650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.322619    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume\") pod \"baefc472-3976-4c96-8bcc-95ac49485bdd\" (UID: \"baefc472-3976-4c96-8bcc-95ac49485bdd\") "
Nov 03 12:32:39.322866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.322839    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume" pod "f14a0f84-2f47-41a9-9b7b-60406dbe48e7" (UID: "f14a0f84-2f47-41a9-9b7b-60406dbe48e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:39.323162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.323139    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:39.323257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.323094    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume" pod "baefc472-3976-4c96-8bcc-95ac49485bdd" (UID: "baefc472-3976-4c96-8bcc-95ac49485bdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:39.324718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.324693    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb" pod "f14a0f84-2f47-41a9-9b7b-60406dbe48e7" (UID: "f14a0f84-2f47-41a9-9b7b-60406dbe48e7"). InnerVolumeSpecName "kube-api-access-jszpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:39.324859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.324831    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv" pod "baefc472-3976-4c96-8bcc-95ac49485bdd" (UID: "baefc472-3976-4c96-8bcc-95ac49485bdd"). InnerVolumeSpecName "kube-api-access-62wxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:39.359124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.359096    4546 generic.go:356] "Generic (PLEG): container finished" podID="f14a0f84-2f47-41a9-9b7b-60406dbe48e7" containerID="4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54" exitCode=0
Nov 03 12:32:39.359274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.359195    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd" event={"ID":"f14a0f84-2f47-41a9-9b7b-60406dbe48e7","Type":"ContainerDied","Data":"4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54"}
Nov 03 12:32:39.359274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.359264    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd" event={"ID":"f14a0f84-2f47-41a9-9b7b-60406dbe48e7","Type":"ContainerDied","Data":"174e71b2b94311d38133516c3476f05964311d1a180a58d372a9d69ccaf1241d"}
Nov 03 12:32:39.359407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.359280    4546 scope.go:122] "RemoveContainer" containerID="4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54"
Nov 03 12:32:39.359407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.359208    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"
Nov 03 12:32:39.370825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.361107    4546 generic.go:356] "Generic (PLEG): container finished" podID="baefc472-3976-4c96-8bcc-95ac49485bdd" containerID="9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3" exitCode=0
Nov 03 12:32:39.370825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.361169    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"
Nov 03 12:32:39.370825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.361178    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" event={"ID":"baefc472-3976-4c96-8bcc-95ac49485bdd","Type":"ContainerDied","Data":"9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3"}
Nov 03 12:32:39.370825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.361505    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f" event={"ID":"baefc472-3976-4c96-8bcc-95ac49485bdd","Type":"ContainerDied","Data":"2f88dfd1b98d3f945ac369244e94dc864996b4e8ea950ea0fe1208369457892a"}
Nov 03 12:32:39.378025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.378006    4546 scope.go:122] "RemoveContainer" containerID="4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54"
Nov 03 12:32:39.378568 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:32:39.378532    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54\": container with ID starting with 4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54 not found: ID does not exist" containerID="4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54"
Nov 03 12:32:39.378657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.378572    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54"} err="failed to get container status \"4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54\": rpc error: code = NotFound desc = could not find container \"4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54\": container with ID starting with 4f964f7c38a979a7071766d125c4be213efd37dd3e9f641c7923a3c6c555ec54 not found: ID does not exist"
Nov 03 12:32:39.378657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.378598    4546 scope.go:122] "RemoveContainer" containerID="9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3"
Nov 03 12:32:39.417279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.417231    4546 scope.go:122] "RemoveContainer" containerID="9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3"
Nov 03 12:32:39.418379 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:32:39.417955    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3\": container with ID starting with 9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3 not found: ID does not exist" containerID="9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3"
Nov 03 12:32:39.418379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.418018    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3"} err="failed to get container status \"9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3\": rpc error: code = NotFound desc = could not find container \"9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3\": container with ID starting with 9e2725dc6c9457b2ae4282577403c67a01bfb718f2f8f25efb297ef8c4cbe3a3 not found: ID does not exist"
Nov 03 12:32:39.427863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.426602    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62wxv\" (UniqueName: \"kubernetes.io/projected/baefc472-3976-4c96-8bcc-95ac49485bdd-kube-api-access-62wxv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:39.427863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.426639    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baefc472-3976-4c96-8bcc-95ac49485bdd-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:39.427863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.426658    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszpb\" (UniqueName: \"kubernetes.io/projected/f14a0f84-2f47-41a9-9b7b-60406dbe48e7-kube-api-access-jszpb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:39.568243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.568206    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"]
Nov 03 12:32:39.574245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.574218    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2daac7-db45cf4b7-dzr7f"]
Nov 03 12:32:39.675226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.675190    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"]
Nov 03 12:32:39.679148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:39.679086    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2daac7-f88868d-zxwfd"]
Nov 03 12:32:40.575455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:40.575418    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baefc472-3976-4c96-8bcc-95ac49485bdd" path="/var/lib/kubelet/pods/baefc472-3976-4c96-8bcc-95ac49485bdd/volumes"
Nov 03 12:32:40.576057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:40.575874    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14a0f84-2f47-41a9-9b7b-60406dbe48e7" path="/var/lib/kubelet/pods/f14a0f84-2f47-41a9-9b7b-60406dbe48e7/volumes"
Nov 03 12:32:43.446988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.446956    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"]
Nov 03 12:32:43.447463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447449    4546 cpu_manager.go:414] "containerMap: removing container" podUID="38f90670-2e44-4225-adbd-356d5fa485b7" containerName="revoke-keys"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447464    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f90670-2e44-4225-adbd-356d5fa485b7" containerName="revoke-keys"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447491    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f14a0f84-2f47-41a9-9b7b-60406dbe48e7" containerName="payload-processing"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447496    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14a0f84-2f47-41a9-9b7b-60406dbe48e7" containerName="payload-processing"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447507    4546 cpu_manager.go:414] "containerMap: removing container" podUID="baefc472-3976-4c96-8bcc-95ac49485bdd" containerName="payload-pre-processing"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447512    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baefc472-3976-4c96-8bcc-95ac49485bdd" containerName="payload-pre-processing"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447521    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy"
Nov 03 12:32:43.447524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447526    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy"
Nov 03 12:32:43.447777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447603    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14a0f84-2f47-41a9-9b7b-60406dbe48e7" containerName="payload-processing"
Nov 03 12:32:43.447777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447614    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="baefc472-3976-4c96-8bcc-95ac49485bdd" containerName="payload-pre-processing"
Nov 03 12:32:43.447777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447620    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6951b33a-2fdd-4df7-9f8e-339c2a2c1b4c" containerName="istio-proxy"
Nov 03 12:32:43.447777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.447630    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f90670-2e44-4225-adbd-356d5fa485b7" containerName="revoke-keys"
Nov 03 12:32:43.452594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.452579    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.552590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.552561    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"]
Nov 03 12:32:43.574546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.574517    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.574724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.574560    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.574826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.574718    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.574895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.574827    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.574895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.574868    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.574962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.574936    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.575032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.575016    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.575078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.575063    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.575152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.575136    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtlp\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676080    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676121    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676143    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676253    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676368    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676424    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676426    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676479    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtlp\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676496    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676589    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676614    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676700    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676726    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.676903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.676896    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.679131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.679105    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.679299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.679280    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.729056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.729031    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.729172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.729091    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtlp\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:43.765833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:43.765816    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:44.192605 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:44.192569    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bca048c_17d5_4693_87cf_df55c4d16ddc.slice/crio-71016a8917d2b4264738e09955d36547384411871dde8c862dbb8edda9908c23 WatchSource:0}: Error finding container 71016a8917d2b4264738e09955d36547384411871dde8c862dbb8edda9908c23: Status 404 returned error can't find the container with id 71016a8917d2b4264738e09955d36547384411871dde8c862dbb8edda9908c23
Nov 03 12:32:44.193004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.192982    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"]
Nov 03 12:32:44.195200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.195163    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:44.195295 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.195248    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:44.195367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.195292    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:44.416277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.416236    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" event={"ID":"5bca048c-17d5-4693-87cf-df55c4d16ddc","Type":"ContainerStarted","Data":"65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7"}
Nov 03 12:32:44.416452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.416295    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" event={"ID":"5bca048c-17d5-4693-87cf-df55c4d16ddc","Type":"ContainerStarted","Data":"71016a8917d2b4264738e09955d36547384411871dde8c862dbb8edda9908c23"}
Nov 03 12:32:44.526141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.526093    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podStartSLOduration=1.5260761619999998 podStartE2EDuration="1.526076162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:44.523240587 +0000 UTC m=+2506.094824517" watchObservedRunningTime="2026-11-03 12:32:44.526076162 +0000 UTC m=+2506.097660075"
Nov 03 12:32:44.753537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.753503    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"]
Nov 03 12:32:44.757760 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.757724    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.766921 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.766902    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:44.809205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.809136    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cacb84-openshift-default-dockercfg-xjqrs\""
Nov 03 12:32:44.819356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.819306    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"]
Nov 03 12:32:44.895348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895315    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895363    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895429    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895462    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895479    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895696    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895735    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gr5\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895835    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.895910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.895899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.997699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997665    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.997933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997708    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gr5\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.997933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997739    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.997933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997781    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.997933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997905    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.998205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997957    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.998205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.997987    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.998205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998017    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.998205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998041    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.998205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998151    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.998205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998186    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.999099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998278    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.999099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998370    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:44.999099 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:44.998698    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:45.000449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.000424    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:45.001013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.000988    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:45.052285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.052257    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gr5\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:45.052588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.052572    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token\") pod \"e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") " pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:45.071776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.071675    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:45.533888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.533855    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"]
Nov 03 12:32:45.556937 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:45.556896    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0956ce2f_6a77_40cc_a5df_f3ce1cfd5a0d.slice/crio-2d68c655ca0b74989711fd870b22657f0eebbeabe70fbeeb4840928e13f00f38 WatchSource:0}: Error finding container 2d68c655ca0b74989711fd870b22657f0eebbeabe70fbeeb4840928e13f00f38: Status 404 returned error can't find the container with id 2d68c655ca0b74989711fd870b22657f0eebbeabe70fbeeb4840928e13f00f38
Nov 03 12:32:45.559688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.559656    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:45.559808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.559723    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:45.559808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.559774    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:45.768010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.767978    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:45.768172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:45.768026    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:46.446440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:46.446402    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" event={"ID":"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d","Type":"ContainerStarted","Data":"332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"}
Nov 03 12:32:46.446631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:46.446457    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" event={"ID":"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d","Type":"ContainerStarted","Data":"2d68c655ca0b74989711fd870b22657f0eebbeabe70fbeeb4840928e13f00f38"}
Nov 03 12:32:46.649572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:46.649523    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podStartSLOduration=2.649509364 podStartE2EDuration="2.649509364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:46.601651049 +0000 UTC m=+2508.173234960" watchObservedRunningTime="2026-11-03 12:32:46.649509364 +0000 UTC m=+2508.221093276"
Nov 03 12:32:46.766814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:46.766785    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:46.766988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:46.766834    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:47.073191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:47.073111    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:47.079510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:47.079482    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:47.453271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:47.453177    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:47.454161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:47.454143    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:32:47.766970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:47.766935    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:47.767452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:47.766993    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:48.064142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.064048    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"]
Nov 03 12:32:48.068716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.068694    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071211    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071250    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071306    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071355    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071378    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071476    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjhv\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071543    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071691    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.071780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.071729    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.144554 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.144169    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bbca30-openshift-default-dockercfg-xx7jx\""
Nov 03 12:32:48.183122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.179438    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"]
Nov 03 12:32:48.185498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185466    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.185630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185537    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.185630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185584    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.185712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185630    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.185712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185662    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.185712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185690    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjhv\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.187929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.186345    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.187929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.186610    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.187929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.187393    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.187929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.185741    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.188149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.187944    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.188149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.187978    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.188653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.188631    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.189414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.189394    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.193235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.192215    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.195004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.194980    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.269772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.266936    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.287646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.287603    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjhv\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.382563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.382479    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:48.766903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.766868    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:48.767091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.766920    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:48.841077 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:32:48.841042    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a0349b_1f17_4df2_9167_6acd005107d1.slice/crio-4d8ce6e1ff551f9a83a91333c6b2d640f802a4d9ff80741763a487ac7e8b5999 WatchSource:0}: Error finding container 4d8ce6e1ff551f9a83a91333c6b2d640f802a4d9ff80741763a487ac7e8b5999: Status 404 returned error can't find the container with id 4d8ce6e1ff551f9a83a91333c6b2d640f802a4d9ff80741763a487ac7e8b5999
Nov 03 12:32:48.841686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.841660    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"]
Nov 03 12:32:48.843998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.843969    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:48.844067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.844032    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:48.844067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.844058    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:48.905084 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.905057    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"]
Nov 03 12:32:48.920502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.919974    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:48.966613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:48.966582    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"]
Nov 03 12:32:49.001106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001077    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001112    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001163    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001182    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zr5\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001207    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001222    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001281    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001305    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.001381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.001329    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.047727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.045554    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a4455f-openshift-default-dockercfg-b2vrp\""
Nov 03 12:32:49.102308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102274    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102317    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102392    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102413    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102462    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102479    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zr5\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102505    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102519    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.102830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.102581    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.103339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.103313    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.103414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.103326    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.103561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.103539    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.103827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.103733    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.103961 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.103893    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.105460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.105431    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.105679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.105664    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.201785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.200705    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zr5\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.214330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.214294    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token\") pod \"e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") " pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.268790 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.264242    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:49.476981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.476947    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" event={"ID":"44a0349b-1f17-4df2-9167-6acd005107d1","Type":"ContainerStarted","Data":"a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"}
Nov 03 12:32:49.477194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.476996    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" event={"ID":"44a0349b-1f17-4df2-9167-6acd005107d1","Type":"ContainerStarted","Data":"4d8ce6e1ff551f9a83a91333c6b2d640f802a4d9ff80741763a487ac7e8b5999"}
Nov 03 12:32:49.594313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.593654    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podStartSLOduration=1.593637972 podStartE2EDuration="1.593637972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:49.592854075 +0000 UTC m=+2511.164438023" watchObservedRunningTime="2026-11-03 12:32:49.593637972 +0000 UTC m=+2511.165221885"
Nov 03 12:32:49.753282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.753247    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"]
Nov 03 12:32:49.779007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.769924    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:49.779007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.770014    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:49.785845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.784085    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:49.785845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.784198    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:49.785845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:49.784242    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:32:50.389777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.386871    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:50.402808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.398126    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:50.612791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.612558    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" event={"ID":"deb4b2db-141c-48d2-935d-11696737f7d6","Type":"ContainerStarted","Data":"69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"}
Nov 03 12:32:50.612791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.612632    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:50.612791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.612661    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" event={"ID":"deb4b2db-141c-48d2-935d-11696737f7d6","Type":"ContainerStarted","Data":"3584245dece089b1a0a4e3bd1d570ca7e7ceafc451bc5321c6c73e504fbe7200"}
Nov 03 12:32:50.612791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.612719    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:32:50.772775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.769119    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:50.772775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:50.769177    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:51.274307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.273974    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:51.282156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.282130    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:51.398374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.398240    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podStartSLOduration=3.398219301 podStartE2EDuration="3.398219301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:50.88428907 +0000 UTC m=+2512.455872984" watchObservedRunningTime="2026-11-03 12:32:51.398219301 +0000 UTC m=+2512.969803214"
Nov 03 12:32:51.627777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.616867    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:51.627777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.619510    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:32:51.769774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.767064    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 12:32:51.769774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:51.767129    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 12:32:53.807928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:53.807884    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:32:53.808564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:53.807943    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:32:53.904800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:53.904767    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:53.905319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:53.905288    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:32:53.906234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:32:53.906213    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:33:07.650805 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:33:07.650770    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:33:13.697530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.697497    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"]
Nov 03 12:33:13.703257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.703237    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.874510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874474    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.874510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874514    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.874822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874599    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.874822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874661    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.874822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874715    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglwl\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.874822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874793    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.875014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874834    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.875014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874868    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.875014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.874983    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.907369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.907343    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"]
Nov 03 12:33:13.976036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976009    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976044    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976091    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976119    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976298    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglwl\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976353    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976409    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976454    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976482    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976500    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976509    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976797    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.976897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.976832    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.977117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.977100    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.978744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.978726    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:13.978858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:13.978811    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:14.029568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.029514    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:14.032054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.032035    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglwl\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:14.329471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.329375    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:14.750638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.750596    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"]
Nov 03 12:33:14.784439 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:33:14.784412    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cf3193_e29c_4cf1_9155_624b3c55af65.slice/crio-daacffe4bb97356d7cfbb813ca7afe74a864f513a9e472ab8a2e1e3a041c0b82 WatchSource:0}: Error finding container daacffe4bb97356d7cfbb813ca7afe74a864f513a9e472ab8a2e1e3a041c0b82: Status 404 returned error can't find the container with id daacffe4bb97356d7cfbb813ca7afe74a864f513a9e472ab8a2e1e3a041c0b82
Nov 03 12:33:14.786872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.786844    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:33:14.786947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.786908    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:33:14.786947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:14.786935    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:33:15.788904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:15.788849    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" event={"ID":"17cf3193-e29c-4cf1-9155-624b3c55af65","Type":"ContainerStarted","Data":"639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb"}
Nov 03 12:33:15.789300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:15.788940    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" event={"ID":"17cf3193-e29c-4cf1-9155-624b3c55af65","Type":"ContainerStarted","Data":"daacffe4bb97356d7cfbb813ca7afe74a864f513a9e472ab8a2e1e3a041c0b82"}
Nov 03 12:33:15.974725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:15.973923    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podStartSLOduration=2.97389922 podStartE2EDuration="2.97389922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:15.945017603 +0000 UTC m=+2537.516601544" watchObservedRunningTime="2026-11-03 12:33:15.97389922 +0000 UTC m=+2537.545483134"
Nov 03 12:33:16.339775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:16.339075    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 12:33:16.339775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:16.339156    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 12:33:16.339775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:16.339549    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:17.331183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:17.331145    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 12:33:17.332561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:17.332527    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 12:33:18.331159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:18.331119    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 12:33:18.331437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:18.331407    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 12:33:19.338774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:19.338163    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 12:33:19.338774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:19.338224    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 12:33:20.332194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:20.332159    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 12:33:20.332390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:20.332217    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 12:33:21.330307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:21.330264    4546 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 12:33:21.330875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:21.330328    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 12:33:22.334340 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:22.334303    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:22.334829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:22.334728    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:22.335344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:22.335319    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:33:23.250902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.250869    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:33:23.251102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.250916    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:33:23.868508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.868426    4546 generic.go:356] "Generic (PLEG): container finished" podID="69594464-d692-4da3-8dd3-d31e710455c8" containerID="ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108" exitCode=0
Nov 03 12:33:23.869003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.868503    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" event={"ID":"69594464-d692-4da3-8dd3-d31e710455c8","Type":"ContainerDied","Data":"ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"}
Nov 03 12:33:23.869170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.869153    4546 scope.go:122] "RemoveContainer" containerID="ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"
Nov 03 12:33:23.870611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.870586    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:33:23.870683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.870651    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:33:23.870720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:23.870682    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:33:24.877190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:24.877159    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" event={"ID":"69594464-d692-4da3-8dd3-d31e710455c8","Type":"ContainerStarted","Data":"04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4"}
Nov 03 12:33:25.274396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:25.274361    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:33:25.274586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:25.274518    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 12:33:25.274586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:25.274568    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 12:33:26.273950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:26.273915    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 12:33:26.274481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:26.273969    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 12:33:27.274148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:27.274113    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 12:33:27.274541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:27.274162    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 12:33:28.273828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:28.273791    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 12:33:28.274010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:28.273850    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 12:33:29.273819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:29.273788    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 12:33:29.274245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:29.273852    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 12:33:30.278873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:30.278845    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:33:30.279378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:30.279362    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:33:30.280106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:30.280087    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 12:33:37.652318 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:33:37.652203    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:33:38.046617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:38.046585    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"]
Nov 03 12:33:38.092620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:38.092591    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7a8e96-mzzw8"]
Nov 03 12:33:38.572811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:38.572784    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5874736c-b1d2-4fc8-9763-4a6bd8f5bdad" path="/var/lib/kubelet/pods/5874736c-b1d2-4fc8-9763-4a6bd8f5bdad/volumes"
Nov 03 12:33:56.077132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:56.072686    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"]
Nov 03 12:33:56.099481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:56.099453    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c29bd-fnvcj"]
Nov 03 12:33:56.572875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:33:56.572848    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9883170b-24e0-4039-811e-c855731c4aae" path="/var/lib/kubelet/pods/9883170b-24e0-4039-811e-c855731c4aae/volumes"
Nov 03 12:34:02.526231 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.526198    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"]
Nov 03 12:34:02.531987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.531965    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.582353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.582324    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8570a0-openshift-default-dockercfg-5hmpq\""
Nov 03 12:34:02.609106 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609071    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609183    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609216    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609277 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609266    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609384    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609467    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckg4k\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609642    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.609946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609924    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.610045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.609977    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.634609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.634579    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"]
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711143    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711228    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711254    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711286    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711351    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711414    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckg4k\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711482    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711532    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.711971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.711596    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.713848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.712143    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.713848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.712903    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.713848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.713019    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.713848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.713429    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.714153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.714049    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.725445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.725378    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.725605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.725581    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.766518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.766485    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.766655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.766539    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckg4k\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k\") pod \"e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q\" (UID: \"689340e2-674e-40de-a499-199fada62980\") " pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:02.846988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:02.846961    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:03.300822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.299017    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"]
Nov 03 12:34:03.309624 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:03.309591    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689340e2_674e_40de_a499_199fada62980.slice/crio-c8c6ee4f82b8fd354e9c93f836acfe226786a79a93af2916e1d0e6faf69be90a WatchSource:0}: Error finding container c8c6ee4f82b8fd354e9c93f836acfe226786a79a93af2916e1d0e6faf69be90a: Status 404 returned error can't find the container with id c8c6ee4f82b8fd354e9c93f836acfe226786a79a93af2916e1d0e6faf69be90a
Nov 03 12:34:03.313546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.313505    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:03.313670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.313593    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:03.313670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.313638    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:03.494455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.494157    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:34:03.494455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.494207    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:34:03.731397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.731360    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"]
Nov 03 12:34:03.736780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.736721    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829215    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829291    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829388    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829428    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829511    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829546    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829577    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829605    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.829740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.829655    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvhjg\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.929437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.929400    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"]
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932422    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvhjg\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932517    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932562    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932599    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932643    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932769    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932821    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932869    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.933421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.932911    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.934411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.934387    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.934616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.934586    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.935055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.935032    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.935219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.935192    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.935501 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.935480    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.938143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.938101    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.938556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.938508    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.987951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.987876    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvhjg\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:03.987951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:03.987911    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:04.054352 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.054319    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:04.248291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.248192    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" event={"ID":"689340e2-674e-40de-a499-199fada62980","Type":"ContainerStarted","Data":"cf3826cbd97bc3f166929793ebf5fa0e3fda9d5e1f98f62c77653b0a00b668c1"}
Nov 03 12:34:04.248291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.248258    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" event={"ID":"689340e2-674e-40de-a499-199fada62980","Type":"ContainerStarted","Data":"c8c6ee4f82b8fd354e9c93f836acfe226786a79a93af2916e1d0e6faf69be90a"}
Nov 03 12:34:04.404463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.404395    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podStartSLOduration=2.404371418 podStartE2EDuration="2.404371418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:04.35527609 +0000 UTC m=+2585.926860001" watchObservedRunningTime="2026-11-03 12:34:04.404371418 +0000 UTC m=+2585.975955331"
Nov 03 12:34:04.511912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.511826    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"]
Nov 03 12:34:04.521822 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:04.521775    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420474ed_cdc6_4c0e_bd58_53dd11cfa678.slice/crio-61aa2dc246ad7880720be6a36f7c9a2371e6d5628dceec7a20848bc18f3ddd58 WatchSource:0}: Error finding container 61aa2dc246ad7880720be6a36f7c9a2371e6d5628dceec7a20848bc18f3ddd58: Status 404 returned error can't find the container with id 61aa2dc246ad7880720be6a36f7c9a2371e6d5628dceec7a20848bc18f3ddd58
Nov 03 12:34:04.524534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.524480    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:04.524660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.524588    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:04.524731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.524667    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:04.847825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.847719    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:04.854499 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:04.854470    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:05.257934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:05.257890    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" event={"ID":"420474ed-cdc6-4c0e-bd58-53dd11cfa678","Type":"ContainerStarted","Data":"b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b"}
Nov 03 12:34:05.258155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:05.257962    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" event={"ID":"420474ed-cdc6-4c0e-bd58-53dd11cfa678","Type":"ContainerStarted","Data":"61aa2dc246ad7880720be6a36f7c9a2371e6d5628dceec7a20848bc18f3ddd58"}
Nov 03 12:34:05.258356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:05.258326    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:05.259363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:05.259342    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:34:05.371436 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:05.371370    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podStartSLOduration=2.371352552 podStartE2EDuration="2.371352552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:05.366045654 +0000 UTC m=+2586.937629566" watchObservedRunningTime="2026-11-03 12:34:05.371352552 +0000 UTC m=+2586.942936463"
Nov 03 12:34:06.054709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:06.054668    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:06.056455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:06.056429    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 12:34:06.056588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:06.056481    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 12:34:06.653911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:06.653829    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:34:06.653911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:06.653887    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:34:07.055356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.055326    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 12:34:07.055780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.055377    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 12:34:07.275944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.275917    4546 generic.go:356] "Generic (PLEG): container finished" podID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerID="332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d" exitCode=0
Nov 03 12:34:07.276118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.275986    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" event={"ID":"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d","Type":"ContainerDied","Data":"332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"}
Nov 03 12:34:07.276450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.276433    4546 scope.go:122] "RemoveContainer" containerID="332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"
Nov 03 12:34:07.277828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.277811    4546 generic.go:356] "Generic (PLEG): container finished" podID="44a0349b-1f17-4df2-9167-6acd005107d1" containerID="a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb" exitCode=0
Nov 03 12:34:07.277953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.277902    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" event={"ID":"44a0349b-1f17-4df2-9167-6acd005107d1","Type":"ContainerDied","Data":"a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"}
Nov 03 12:34:07.278232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.278196    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:07.278350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.278292    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:07.278350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.278338    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:07.278510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.278493    4546 scope.go:122] "RemoveContainer" containerID="a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"
Nov 03 12:34:07.279940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.279916    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:07.280004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.279991    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:07.280049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:07.280035    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:07.674682 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:34:07.674582    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:34:08.057774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.056077    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 12:34:08.057774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.056136    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 12:34:08.291776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.291716    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" event={"ID":"44a0349b-1f17-4df2-9167-6acd005107d1","Type":"ContainerStarted","Data":"3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef"}
Nov 03 12:34:08.293965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.293943    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" event={"ID":"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d","Type":"ContainerStarted","Data":"746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0"}
Nov 03 12:34:08.384051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.383768    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:34:08.384051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.383902    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 12:34:08.384051 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:08.383947    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 12:34:09.060777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.060171    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 12:34:09.060777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.060238    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 12:34:09.080785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.077846    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:34:09.080785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.078005    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 12:34:09.080785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.078087    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 12:34:09.383328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.383229    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 12:34:09.383617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:09.383571    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 12:34:10.063777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:10.063066    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 12:34:10.063777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:10.063141    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 12:34:10.080782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:10.079291    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 12:34:10.080782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:10.079352    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 12:34:10.383266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:10.383166    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 12:34:10.383266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:10.383223    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 12:34:11.096582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.089820    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 12:34:11.096582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.089941    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 12:34:11.098904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.098678    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:11.099665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.099648    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:11.102484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.102459    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:34:11.290607 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.290574    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"]
Nov 03 12:34:11.295686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.295667    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.348108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.348029    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8570a0\""
Nov 03 12:34:11.348451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.348431    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8570a0-dockercfg-cfg4b\""
Nov 03 12:34:11.378256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.378208    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"]
Nov 03 12:34:11.383744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.383715    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 12:34:11.383887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.383808    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 12:34:11.424777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.414707    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.469183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.469143    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzz56\" (UniqueName: \"kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56\") pod \"payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.471733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.471705    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"]
Nov 03 12:34:11.475940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.475919    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.492224 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.492199    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"]
Nov 03 12:34:11.717828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.695229    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzz56\" (UniqueName: \"kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56\") pod \"payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.717828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.695793    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.717828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.696253    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.717828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.696294    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br45z\" (UniqueName: \"kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z\") pod \"payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.717828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.716500    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.778972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.778906    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzz56\" (UniqueName: \"kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56\") pod \"payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:11.798223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.798185    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.798376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.798287    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br45z\" (UniqueName: \"kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z\") pod \"payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.798896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.798867    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.957592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.957557    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br45z\" (UniqueName: \"kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z\") pod \"payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:11.958278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:11.958019    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:12.078912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.078153    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 12:34:12.078912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.078221    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 12:34:12.094556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.094513    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:12.383416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.383377    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 12:34:12.383828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.383445    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 12:34:12.489557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.489526    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"]
Nov 03 12:34:12.490300 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:12.490275    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd012d0_647c_4be4_a45b_0a2504fe9143.slice/crio-e154d36e5a0c69dc5a69a54f52c500d09aa4072ddbf741e41992cb51fd27f2a6 WatchSource:0}: Error finding container e154d36e5a0c69dc5a69a54f52c500d09aa4072ddbf741e41992cb51fd27f2a6: Status 404 returned error can't find the container with id e154d36e5a0c69dc5a69a54f52c500d09aa4072ddbf741e41992cb51fd27f2a6
Nov 03 12:34:12.591818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:12.591509    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"]
Nov 03 12:34:12.593793 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:12.593522    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6d02ad_1e47_42d7_b90c_00e5ad9163d9.slice/crio-b9e0978003ec060e8193b03be7164e7f8e0a62f5521a2fff1eacd581bbbb4116 WatchSource:0}: Error finding container b9e0978003ec060e8193b03be7164e7f8e0a62f5521a2fff1eacd581bbbb4116: Status 404 returned error can't find the container with id b9e0978003ec060e8193b03be7164e7f8e0a62f5521a2fff1eacd581bbbb4116
Nov 03 12:34:13.064972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.064803    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"]
Nov 03 12:34:13.066926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.066893    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5710-w26g7"]
Nov 03 12:34:13.072243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.072218    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 12:34:13.072353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.072262    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 12:34:13.377918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.377827    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" event={"ID":"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9","Type":"ContainerStarted","Data":"f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772"}
Nov 03 12:34:13.378430 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.378394    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" event={"ID":"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9","Type":"ContainerStarted","Data":"b9e0978003ec060e8193b03be7164e7f8e0a62f5521a2fff1eacd581bbbb4116"}
Nov 03 12:34:13.378509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.378481    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:13.379445 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.379419    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" event={"ID":"ecd012d0-647c-4be4-a45b-0a2504fe9143","Type":"ContainerStarted","Data":"818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c"}
Nov 03 12:34:13.379570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.379467    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" event={"ID":"ecd012d0-647c-4be4-a45b-0a2504fe9143","Type":"ContainerStarted","Data":"e154d36e5a0c69dc5a69a54f52c500d09aa4072ddbf741e41992cb51fd27f2a6"}
Nov 03 12:34:13.379570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.379519    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:13.382950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.382931    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 12:34:13.383002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.382974    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 12:34:13.504036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:13.503991    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" podStartSLOduration=2.503976552 podStartE2EDuration="2.503976552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:13.499415773 +0000 UTC m=+2595.070999701" watchObservedRunningTime="2026-11-03 12:34:13.503976552 +0000 UTC m=+2595.075560462"
Nov 03 12:34:14.077014 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.076973    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:34:14.077273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.077254    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:34:14.077980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.077962    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 12:34:14.235352 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.235300    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" podStartSLOduration=3.235283683 podStartE2EDuration="3.235283683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:13.611939598 +0000 UTC m=+2595.183523510" watchObservedRunningTime="2026-11-03 12:34:14.235283683 +0000 UTC m=+2595.806867594"
Nov 03 12:34:14.387716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.387646    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:34:14.388173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.388154    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:34:14.388717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.388700    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:34:14.579673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:14.579644    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed798a56-0953-400e-9364-fba55a30d4c4" path="/var/lib/kubelet/pods/ed798a56-0953-400e-9364-fba55a30d4c4/volumes"
Nov 03 12:34:16.079280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.079226    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"]
Nov 03 12:34:16.084220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.084201    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.121821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.121780    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.121914    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122001 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.121993    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.122083    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.122116    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.122255    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.122294    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k89l\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.122485    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.122851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.122517    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.132862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.132828    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-743d9b-openshift-default-dockercfg-7vthz\""
Nov 03 12:34:16.184688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.184653    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"]
Nov 03 12:34:16.224946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.224883    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225235    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225317    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225403    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225458    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225539    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225575    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225618    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225653    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k89l\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.225711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.225691    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.226147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.226120    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.226272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.226254    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.227483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.227454    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.228094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.228065    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.231146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.231120    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.245678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.242049    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.282922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.282891    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k89l\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.283091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.282978    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token\") pod \"e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") " pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.408004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.403143    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:16.833734 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:16.833705    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dae7f1a_c57a_4707_8f78_03a0464be8f6.slice/crio-dcc43d80f4338d49a70013642c3b19c979b8d4e46bacb137c670c9d78513a6c0 WatchSource:0}: Error finding container dcc43d80f4338d49a70013642c3b19c979b8d4e46bacb137c670c9d78513a6c0: Status 404 returned error can't find the container with id dcc43d80f4338d49a70013642c3b19c979b8d4e46bacb137c670c9d78513a6c0
Nov 03 12:34:16.834407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.834372    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"]
Nov 03 12:34:16.836401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.836367    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:16.836517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.836443    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:16.836517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:16.836489    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:17.417060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:17.417026    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" event={"ID":"5dae7f1a-c57a-4707-8f78-03a0464be8f6","Type":"ContainerStarted","Data":"b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"}
Nov 03 12:34:17.417469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:17.417076    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" event={"ID":"5dae7f1a-c57a-4707-8f78-03a0464be8f6","Type":"ContainerStarted","Data":"dcc43d80f4338d49a70013642c3b19c979b8d4e46bacb137c670c9d78513a6c0"}
Nov 03 12:34:17.577741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:17.577680    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podStartSLOduration=1.577662547 podStartE2EDuration="1.577662547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:17.528162339 +0000 UTC m=+2599.099746270" watchObservedRunningTime="2026-11-03 12:34:17.577662547 +0000 UTC m=+2599.149246461"
Nov 03 12:34:18.403506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:18.403472    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:18.409313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:18.409290    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:18.423730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:18.423707    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:18.424636 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:18.424617    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:34:23.982885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:23.982849    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"]
Nov 03 12:34:24.003649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.003614    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.105780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.103991    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-743d9b-dockercfg-4729h\""
Nov 03 12:34:24.105780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.104057    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-743d9b\""
Nov 03 12:34:24.105780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.104621    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"]
Nov 03 12:34:24.138381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.137808    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.138381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.137895    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjzh7\" (UniqueName: \"kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7\") pod \"payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.238964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.238863    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.238964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.238942    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjzh7\" (UniqueName: \"kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7\") pod \"payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.239804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.239776    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.292112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.292078    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjzh7\" (UniqueName: \"kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7\") pod \"payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.390546 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.390515    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 12:34:24.403205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.403183    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:24.794009 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:24.793973    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4dc0d7_09a8_453a_94c0_e2e5a02a5956.slice/crio-dd0187b89f8489cd32973beff68ffd73217dda13b303ecbb384520e8f4a962e6 WatchSource:0}: Error finding container dd0187b89f8489cd32973beff68ffd73217dda13b303ecbb384520e8f4a962e6: Status 404 returned error can't find the container with id dd0187b89f8489cd32973beff68ffd73217dda13b303ecbb384520e8f4a962e6
Nov 03 12:34:24.794410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:24.794386    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"]
Nov 03 12:34:25.397153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:25.397126    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 12:34:25.472035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:25.471999    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" event={"ID":"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956","Type":"ContainerStarted","Data":"6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273"}
Nov 03 12:34:25.472223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:25.472061    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" event={"ID":"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956","Type":"ContainerStarted","Data":"dd0187b89f8489cd32973beff68ffd73217dda13b303ecbb384520e8f4a962e6"}
Nov 03 12:34:25.472223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:25.472096    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:25.630775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:25.608020    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" podStartSLOduration=2.608000332 podStartE2EDuration="2.608000332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:25.606905439 +0000 UTC m=+2607.178489351" watchObservedRunningTime="2026-11-03 12:34:25.608000332 +0000 UTC m=+2607.179584244"
Nov 03 12:34:28.043815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:28.043778    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"]
Nov 03 12:34:28.089251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:28.089223    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e20bba5-sgxzl"]
Nov 03 12:34:28.577182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:28.577144    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732af0de-28b2-46ab-8dd2-7193e46c584c" path="/var/lib/kubelet/pods/732af0de-28b2-46ab-8dd2-7193e46c584c/volumes"
Nov 03 12:34:31.539928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:31.539898    4546 generic.go:356] "Generic (PLEG): container finished" podID="deb4b2db-141c-48d2-935d-11696737f7d6" containerID="69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1" exitCode=0
Nov 03 12:34:31.540357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:31.539971    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" event={"ID":"deb4b2db-141c-48d2-935d-11696737f7d6","Type":"ContainerDied","Data":"69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"}
Nov 03 12:34:31.540409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:31.540397    4546 scope.go:122] "RemoveContainer" containerID="69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"
Nov 03 12:34:31.542153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:31.542126    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:31.542244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:31.542197    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:31.542244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:31.542228    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:32.549738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:32.549698    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" event={"ID":"deb4b2db-141c-48d2-935d-11696737f7d6","Type":"ContainerStarted","Data":"1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64"}
Nov 03 12:34:33.270282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:33.270253    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:34:33.272238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:33.272207    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 12:34:33.272395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:33.272270    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 12:34:34.270005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:34.269971    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 12:34:34.270409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:34.270023    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 12:34:35.269819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:35.269790    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 12:34:35.270017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:35.269842    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 12:34:36.269550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:36.269516    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 12:34:36.270012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:36.269569    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 12:34:36.482512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:36.482480    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 12:34:36.640082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:36.637915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:34:37.269855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:37.269825    4546 patch_prober.go:29] interesting pod/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 12:34:37.270274 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:37.269875    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 12:34:37.663630 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:34:37.663537    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:34:38.344283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:38.344252    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:34:38.346528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:38.346499    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:34:44.044996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:44.044953    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"]
Nov 03 12:34:44.090893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:44.090860    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d058b2c-kr6j6"]
Nov 03 12:34:44.574341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:44.574307    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd859100-f110-4739-84ae-ebedca482be9" path="/var/lib/kubelet/pods/dd859100-f110-4739-84ae-ebedca482be9/volumes"
Nov 03 12:34:46.483115 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.483080    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"]
Nov 03 12:34:46.488733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.488715    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.591773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.587150    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e0615c\""
Nov 03 12:34:46.591773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.587317    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e0615c-dockercfg-vktn9\""
Nov 03 12:34:46.613769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.611768    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp68\" (UniqueName: \"kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68\") pod \"payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.613769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.611895    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.687711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.687642    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"]
Nov 03 12:34:46.689932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.689693    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"]
Nov 03 12:34:46.698518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.698490    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:46.713649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.712938    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp68\" (UniqueName: \"kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68\") pod \"payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.713649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.713080    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.714018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.713998    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.722503 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.722477    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"]
Nov 03 12:34:46.766842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.766771    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp68\" (UniqueName: \"kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68\") pod \"payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.802248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.802210    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:46.814780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.814733    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:46.815028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.815010    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7b9\" (UniqueName: \"kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9\") pod \"payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:46.917466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.916804    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:46.917466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.916945    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7b9\" (UniqueName: \"kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9\") pod \"payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:46.918154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.918112    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:46.971766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:46.971728    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7b9\" (UniqueName: \"kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9\") pod \"payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:47.016514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.016491    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:47.178233 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:47.178205    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc11dafd_0e40_4c14_9bd6_9d6880b94bb6.slice/crio-ebe37801e8b99179d0a19835f273c087c57516329f24b8f9f909cf7eb429dcca WatchSource:0}: Error finding container ebe37801e8b99179d0a19835f273c087c57516329f24b8f9f909cf7eb429dcca: Status 404 returned error can't find the container with id ebe37801e8b99179d0a19835f273c087c57516329f24b8f9f909cf7eb429dcca
Nov 03 12:34:47.179917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.179894    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"]
Nov 03 12:34:47.386946 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:47.386911    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4c2507_6e9f_4043_813c_a90c4aa1cd5d.slice/crio-df64af9217410856e9080d3bfcad72682b1d2edb3086e2d19da520689a14bb86 WatchSource:0}: Error finding container df64af9217410856e9080d3bfcad72682b1d2edb3086e2d19da520689a14bb86: Status 404 returned error can't find the container with id df64af9217410856e9080d3bfcad72682b1d2edb3086e2d19da520689a14bb86
Nov 03 12:34:47.388087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.388062    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"]
Nov 03 12:34:47.697388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.697283    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" event={"ID":"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6","Type":"ContainerStarted","Data":"ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee"}
Nov 03 12:34:47.697388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.697349    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:47.697388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.697370    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" event={"ID":"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6","Type":"ContainerStarted","Data":"ebe37801e8b99179d0a19835f273c087c57516329f24b8f9f909cf7eb429dcca"}
Nov 03 12:34:47.699130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.699099    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" event={"ID":"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d","Type":"ContainerStarted","Data":"b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce"}
Nov 03 12:34:47.699266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.699156    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" event={"ID":"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d","Type":"ContainerStarted","Data":"df64af9217410856e9080d3bfcad72682b1d2edb3086e2d19da520689a14bb86"}
Nov 03 12:34:47.699266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.699186    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:34:47.804561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.804508    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" podStartSLOduration=1.804492116 podStartE2EDuration="1.804492116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:47.803032942 +0000 UTC m=+2629.374616870" watchObservedRunningTime="2026-11-03 12:34:47.804492116 +0000 UTC m=+2629.376076030"
Nov 03 12:34:47.909690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:47.909647    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" podStartSLOduration=1.909634601 podStartE2EDuration="1.909634601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:47.907816328 +0000 UTC m=+2629.479400255" watchObservedRunningTime="2026-11-03 12:34:47.909634601 +0000 UTC m=+2629.481218513"
Nov 03 12:34:49.064741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.064709    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"]
Nov 03 12:34:49.069157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.069123    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152164    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152232    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152444    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152483    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152538    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152627    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vmn\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152821    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152865    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.152927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.152908    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.253878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.253844    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.253897    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.253942    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.253976    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vmn\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254058    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254082    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254129    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254263    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254279    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254310    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254377    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254464    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254545    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.254801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.254688    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.256378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.256356    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.256842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.256823    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.263183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.262894    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"]
Nov 03 12:34:49.306843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.306815    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.307549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.307522    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vmn\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.389031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.388947    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:49.804671 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:34:49.804640    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a45b90_0b75_4a20_837a_8f364ea4cd28.slice/crio-0ee80458f804324c1adfc16ce58f04da40a30b496e3cf226c55f3241eb7b08d3 WatchSource:0}: Error finding container 0ee80458f804324c1adfc16ce58f04da40a30b496e3cf226c55f3241eb7b08d3: Status 404 returned error can't find the container with id 0ee80458f804324c1adfc16ce58f04da40a30b496e3cf226c55f3241eb7b08d3
Nov 03 12:34:49.804882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.804857    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"]
Nov 03 12:34:49.808693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.808660    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:49.808792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.808730    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:49.808792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:49.808780    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:50.739809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:50.739770    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" event={"ID":"65a45b90-0b75-4a20-837a-8f364ea4cd28","Type":"ContainerStarted","Data":"75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f"}
Nov 03 12:34:50.740200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:50.739827    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" event={"ID":"65a45b90-0b75-4a20-837a-8f364ea4cd28","Type":"ContainerStarted","Data":"0ee80458f804324c1adfc16ce58f04da40a30b496e3cf226c55f3241eb7b08d3"}
Nov 03 12:34:50.946869 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:50.946816    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podStartSLOduration=1.946801036 podStartE2EDuration="1.946801036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:50.898099908 +0000 UTC m=+2632.469683847" watchObservedRunningTime="2026-11-03 12:34:50.946801036 +0000 UTC m=+2632.518384946"
Nov 03 12:34:51.389902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:51.389801    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:51.391427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:51.391406    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 12:34:51.391480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:51.391452    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 12:34:52.389931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:52.389898    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 12:34:52.390364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:52.389952    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 12:34:53.001965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.001931    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"]
Nov 03 12:34:53.006407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.006388    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.063220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.057895    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3176d9-openshift-default-dockercfg-cctm5\""
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.244733    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.244824    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.244899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgpw5\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.245044    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.245083    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.245112    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.245160    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.254831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.245201    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.293731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.268066    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.361173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.360838    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"]
Nov 03 12:34:53.369257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369230    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369292    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369332    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369391    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgpw5\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369500    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369534    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369635 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369559    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369635 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.369761 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.369635    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.370184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.370152    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.370313 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.370232    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.370611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.370539    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.370611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.370598    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.370876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.370851    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.373195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.373147    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.373306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.373246    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.389452 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.389424    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 12:34:53.389557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.389465    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 12:34:53.434230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.434196    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.434592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.434298    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgpw5\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5\") pod \"e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") " pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.530341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.529951    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:53.955494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.955444    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"]
Nov 03 12:34:53.972531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.972494    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:53.972682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.972572    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:53.972682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:53.972603    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:34:54.405333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:54.405247    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 12:34:54.405333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:54.405308    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 12:34:54.772688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:54.772655    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" event={"ID":"defce8d6-9532-451e-8ffc-7335e035cf83","Type":"ContainerStarted","Data":"24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"}
Nov 03 12:34:54.773204 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:54.772717    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" event={"ID":"defce8d6-9532-451e-8ffc-7335e035cf83","Type":"ContainerStarted","Data":"cb4d7e27b9fa213c0e76f40538d8fa054fb7b2d87a59089241f5a76f7de07149"}
Nov 03 12:34:54.891489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:54.891431    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podStartSLOduration=2.8914139629999998 podStartE2EDuration="2.891413963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:54.887632842 +0000 UTC m=+2636.459216754" watchObservedRunningTime="2026-11-03 12:34:54.891413963 +0000 UTC m=+2636.462997883"
Nov 03 12:34:55.392993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:55.392956    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 12:34:55.393276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:55.393249    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 12:34:55.531778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:55.531072    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:55.537930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:55.537904    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:55.779839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:55.779812    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:55.780547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:55.780530    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:34:56.394257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:56.394228    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:56.394682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:56.394646    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:56.395375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:56.395360    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:34:58.708942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:58.708907    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 12:34:58.709549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:34:58.708992    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 12:35:00.044352 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:00.044314    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"]
Nov 03 12:35:00.090333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:00.090301    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddb7aa-tddx2"]
Nov 03 12:35:00.594135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:00.594106    4546 scope.go:122] "RemoveContainer" containerID="7e5024cb2c0e957dbc417782c4f81d2ac15e3b80ae2b4baba939d0453e6cec0c"
Nov 03 12:35:00.599890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:00.599860    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f696ab-ccb6-4714-a8f3-42cfcc4593b4" path="/var/lib/kubelet/pods/a2f696ab-ccb6-4714-a8f3-42cfcc4593b4/volumes"
Nov 03 12:35:00.606183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:00.606079    4546 scope.go:122] "RemoveContainer" containerID="fa9d912967100db5fa96f4059d6b78c970bdd1f74904f42d82cd6f8aa5422d2c"
Nov 03 12:35:07.651991 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:35:07.651897    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:35:23.045415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:23.045381    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"]
Nov 03 12:35:23.091400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:23.091368    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22192dcc-dt6lk"]
Nov 03 12:35:24.574078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:24.574045    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cded93-9084-442c-8191-2a6781908c95" path="/var/lib/kubelet/pods/d3cded93-9084-442c-8191-2a6781908c95/volumes"
Nov 03 12:35:27.044821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:27.044776    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"]
Nov 03 12:35:27.091637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:27.091602    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e400204-cvp9s"]
Nov 03 12:35:28.574510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:28.574477    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25784e49-ebd6-43c9-ad16-0c1d200ded42" path="/var/lib/kubelet/pods/25784e49-ebd6-43c9-ad16-0c1d200ded42/volumes"
Nov 03 12:35:37.045010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:37.044922    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"]
Nov 03 12:35:37.090435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:37.090404    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45378f68-6fd54"]
Nov 03 12:35:37.663172 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:35:37.663134    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:35:38.574304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:38.574262    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295" path="/var/lib/kubelet/pods/a4557a45-fdf5-4e0c-bf62-2ca5cd4d5295/volumes"
Nov 03 12:35:41.223876 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:41.223846    4546 generic.go:356] "Generic (PLEG): container finished" podID="689340e2-674e-40de-a499-199fada62980" containerID="cf3826cbd97bc3f166929793ebf5fa0e3fda9d5e1f98f62c77653b0a00b668c1" exitCode=0
Nov 03 12:35:41.224319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:41.223898    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" event={"ID":"689340e2-674e-40de-a499-199fada62980","Type":"ContainerDied","Data":"cf3826cbd97bc3f166929793ebf5fa0e3fda9d5e1f98f62c77653b0a00b668c1"}
Nov 03 12:35:41.224363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:41.224345    4546 scope.go:122] "RemoveContainer" containerID="cf3826cbd97bc3f166929793ebf5fa0e3fda9d5e1f98f62c77653b0a00b668c1"
Nov 03 12:35:41.226293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:41.226260    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:35:41.226418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:41.226334    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:35:41.226418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:41.226367    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:35:42.240537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:42.240499    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" event={"ID":"689340e2-674e-40de-a499-199fada62980","Type":"ContainerStarted","Data":"de675847c2663111f6e86a4df3f033bd0d53a2c5e94bb68d9f275113ea0faaa5"}
Nov 03 12:35:42.847146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:42.847115    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:35:42.847369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:42.847337    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 03 12:35:42.847438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:42.847384    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 03 12:35:43.216850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.206772    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"]
Nov 03 12:35:43.230267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.218568    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" containerID="cri-o://3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef" gracePeriod=30
Nov 03 12:35:43.254210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.253319    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"]
Nov 03 12:35:43.254906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.254875    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy" containerID="cri-o://b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b" gracePeriod=30
Nov 03 12:35:43.260536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.260505    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"]
Nov 03 12:35:43.260776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.260736    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy" containerID="cri-o://75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f" gracePeriod=30
Nov 03 12:35:43.848132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.848100    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 03 12:35:43.848298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:43.848151    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 03 12:35:44.390230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:44.390194    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:35:44.390678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:44.390252    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:35:44.848008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:44.847976    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 03 12:35:44.848195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:44.848031    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 03 12:35:45.848229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:45.848189    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 03 12:35:45.848670 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:45.848271    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 03 12:35:46.848289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:46.848257    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 03 12:35:46.848740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:46.848319    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 03 12:35:47.853192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:47.853158    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:35:47.853700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:47.853686    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:35:47.854480 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:47.854458    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 12:35:48.602801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.602766    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:35:48.634807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.634781    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:35:48.638496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.638472    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:35:48.798621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798510    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.798621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798566    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.798621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798595    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798699    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798732    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798778    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798841    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798885    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798910    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798965    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.798976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.798963    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.799412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799026    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.799412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799061    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.799412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799112    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.799412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799364    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.799681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799627    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.799832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799768    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.799918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799889    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.799998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799935    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.799998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.799973    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.800161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800010    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.800161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800048    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.800161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800119    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.800161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800140    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800200    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800236    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800267    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800295    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800376    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800412    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800408    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.800463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800443    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo\") pod \"44a0349b-1f17-4df2-9167-6acd005107d1\" (UID: \"44a0349b-1f17-4df2-9167-6acd005107d1\") "
Nov 03 12:35:48.800902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800516    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo\") pod \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\" (UID: \"420474ed-cdc6-4c0e-bd58-53dd11cfa678\") "
Nov 03 12:35:48.800902 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.800585    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket\") pod \"65a45b90-0b75-4a20-837a-8f364ea4cd28\" (UID: \"65a45b90-0b75-4a20-837a-8f364ea4cd28\") "
Nov 03 12:35:48.801694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801666    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0349b-1f17-4df2-9167-6acd005107d1-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.801817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801698    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.801817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801713    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.801817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801727    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.801817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801741    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.801817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801773    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.801817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.801786    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.802273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.802247    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.803426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.803395    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "kube-api-access-lvhjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.804686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.804306    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.804686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.804430    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.804686 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.804481    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.805019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.804996    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.805329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.805309    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.805449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.805337    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.805697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.805566    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.805956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.805933    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.806724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.806195    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.806724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.806401    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "kube-api-access-czjhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.806724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.806611    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.806724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.806612    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "kube-api-access-f5vmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.807025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.806958    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.807087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.807017    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data" pod "44a0349b-1f17-4df2-9167-6acd005107d1" (UID: "44a0349b-1f17-4df2-9167-6acd005107d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.807087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.807056    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.807615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.807552    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.807828 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.807801    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy" pod "65a45b90-0b75-4a20-837a-8f364ea4cd28" (UID: "65a45b90-0b75-4a20-837a-8f364ea4cd28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.808652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.808627    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo" pod "420474ed-cdc6-4c0e-bd58-53dd11cfa678" (UID: "420474ed-cdc6-4c0e-bd58-53dd11cfa678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.903509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903470    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903505    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czjhv\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-kube-api-access-czjhv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903518    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903528    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903536    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903545    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vmn\" (UniqueName: \"kubernetes.io/projected/65a45b90-0b75-4a20-837a-8f364ea4cd28-kube-api-access-f5vmn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903554    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903562    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvhjg\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-kube-api-access-lvhjg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903570    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903578    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903587    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420474ed-cdc6-4c0e-bd58-53dd11cfa678-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903594    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a45b90-0b75-4a20-837a-8f364ea4cd28-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903602    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0349b-1f17-4df2-9167-6acd005107d1-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903623    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903632    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903640    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903648    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0349b-1f17-4df2-9167-6acd005107d1-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903656    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0349b-1f17-4df2-9167-6acd005107d1-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903664    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420474ed-cdc6-4c0e-bd58-53dd11cfa678-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.903998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:48.903672    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a45b90-0b75-4a20-837a-8f364ea4cd28-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:49.321645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.321610    4546 generic.go:356] "Generic (PLEG): container finished" podID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerID="b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b" exitCode=0
Nov 03 12:35:49.321881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.321654    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" event={"ID":"420474ed-cdc6-4c0e-bd58-53dd11cfa678","Type":"ContainerDied","Data":"b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b"}
Nov 03 12:35:49.321881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.321711    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"
Nov 03 12:35:49.321881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.321765    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf" event={"ID":"420474ed-cdc6-4c0e-bd58-53dd11cfa678","Type":"ContainerDied","Data":"61aa2dc246ad7880720be6a36f7c9a2371e6d5628dceec7a20848bc18f3ddd58"}
Nov 03 12:35:49.321881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.321802    4546 scope.go:122] "RemoveContainer" containerID="b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b"
Nov 03 12:35:49.324328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.324297    4546 generic.go:356] "Generic (PLEG): container finished" podID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerID="75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f" exitCode=0
Nov 03 12:35:49.324442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.324334    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" event={"ID":"65a45b90-0b75-4a20-837a-8f364ea4cd28","Type":"ContainerDied","Data":"75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f"}
Nov 03 12:35:49.324442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.324381    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll" event={"ID":"65a45b90-0b75-4a20-837a-8f364ea4cd28","Type":"ContainerDied","Data":"0ee80458f804324c1adfc16ce58f04da40a30b496e3cf226c55f3241eb7b08d3"}
Nov 03 12:35:49.324442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.324385    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"
Nov 03 12:35:49.326307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.326286    4546 generic.go:356] "Generic (PLEG): container finished" podID="44a0349b-1f17-4df2-9167-6acd005107d1" containerID="3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef" exitCode=0
Nov 03 12:35:49.327063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.327048    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"
Nov 03 12:35:49.328634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.328611    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" event={"ID":"44a0349b-1f17-4df2-9167-6acd005107d1","Type":"ContainerDied","Data":"3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef"}
Nov 03 12:35:49.328788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.328665    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd" event={"ID":"44a0349b-1f17-4df2-9167-6acd005107d1","Type":"ContainerDied","Data":"4d8ce6e1ff551f9a83a91333c6b2d640f802a4d9ff80741763a487ac7e8b5999"}
Nov 03 12:35:49.337154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.337134    4546 scope.go:122] "RemoveContainer" containerID="b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b"
Nov 03 12:35:49.337468 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:35:49.337447    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b\": container with ID starting with b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b not found: ID does not exist" containerID="b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b"
Nov 03 12:35:49.337555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.337477    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b"} err="failed to get container status \"b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b\": rpc error: code = NotFound desc = could not find container \"b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b\": container with ID starting with b71c50bea2157c5a4ccd9ce30098e3cff903c6e536d7a2841a5aabd72182d44b not found: ID does not exist"
Nov 03 12:35:49.337555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.337495    4546 scope.go:122] "RemoveContainer" containerID="75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f"
Nov 03 12:35:49.347823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.347801    4546 scope.go:122] "RemoveContainer" containerID="75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f"
Nov 03 12:35:49.348141 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:35:49.348122    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f\": container with ID starting with 75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f not found: ID does not exist" containerID="75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f"
Nov 03 12:35:49.348212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.348149    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f"} err="failed to get container status \"75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f\": rpc error: code = NotFound desc = could not find container \"75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f\": container with ID starting with 75a0a60301afcfaf2332c78fec773c0952e90e90cc4422ee4a868f23a22c9c7f not found: ID does not exist"
Nov 03 12:35:49.348212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.348168    4546 scope.go:122] "RemoveContainer" containerID="3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef"
Nov 03 12:35:49.358319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.358298    4546 scope.go:122] "RemoveContainer" containerID="a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"
Nov 03 12:35:49.368456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.368437    4546 scope.go:122] "RemoveContainer" containerID="3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef"
Nov 03 12:35:49.368842 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:35:49.368809    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef\": container with ID starting with 3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef not found: ID does not exist" containerID="3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef"
Nov 03 12:35:49.368960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.368842    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef"} err="failed to get container status \"3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef\": rpc error: code = NotFound desc = could not find container \"3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef\": container with ID starting with 3040861166f9033a9140267418454fe986d4af99dd5f76e53148551fb90c52ef not found: ID does not exist"
Nov 03 12:35:49.368960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.368861    4546 scope.go:122] "RemoveContainer" containerID="a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"
Nov 03 12:35:49.369125 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:35:49.369108    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb\": container with ID starting with a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb not found: ID does not exist" containerID="a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"
Nov 03 12:35:49.369171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.369133    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb"} err="failed to get container status \"a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb\": rpc error: code = NotFound desc = could not find container \"a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb\": container with ID starting with a1e40b9e5cb5fe53f91ef371240a75919aeebdb36caffc83ffcc7123dfecdbfb not found: ID does not exist"
Nov 03 12:35:49.477885 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.477847    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"]
Nov 03 12:35:49.523677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.523596    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-4t5jf"]
Nov 03 12:35:49.582618 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.582577    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"]
Nov 03 12:35:49.586702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.586671    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-tthmd"]
Nov 03 12:35:49.687736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.687703    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"]
Nov 03 12:35:49.691555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:49.691531    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-xxpll"]
Nov 03 12:35:50.573909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:50.573879    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" path="/var/lib/kubelet/pods/420474ed-cdc6-4c0e-bd58-53dd11cfa678/volumes"
Nov 03 12:35:50.574393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:50.574379    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" path="/var/lib/kubelet/pods/44a0349b-1f17-4df2-9167-6acd005107d1/volumes"
Nov 03 12:35:50.574899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:50.574883    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" path="/var/lib/kubelet/pods/65a45b90-0b75-4a20-837a-8f364ea4cd28/volumes"
Nov 03 12:35:58.045091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:58.045052    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"]
Nov 03 12:35:58.091953 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:58.091922    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217d3cbd-8xd2f"]
Nov 03 12:35:58.625417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:35:58.624000    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0facd28e-cf85-433d-a4f6-c102cc8862ca" path="/var/lib/kubelet/pods/0facd28e-cf85-433d-a4f6-c102cc8862ca/volumes"
Nov 03 12:36:00.642820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:00.642790    4546 scope.go:122] "RemoveContainer" containerID="aa07609eef0a7ad5b15c11ded1d04542ab9bcd768c6fcdf9b145bc28fb6c7b62"
Nov 03 12:36:00.653117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:00.653090    4546 scope.go:122] "RemoveContainer" containerID="adb1fda735947277a340b64a080c1bf90a0cacc7cfcfc8b1c24dee7417222a24"
Nov 03 12:36:00.662010 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:00.661989    4546 scope.go:122] "RemoveContainer" containerID="c7eb306f398d5abf760b0a4b11d1ce18326d946ea4d1ec821d98558227a20d8e"
Nov 03 12:36:00.670820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:00.670800    4546 scope.go:122] "RemoveContainer" containerID="8c6e614658fcc8c07c03ad08dc9a7b0061eb6502423b955ef305dc226e4b2be1"
Nov 03 12:36:07.652349 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:36:07.652299    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:36:08.492669 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:08.492633    4546 generic.go:356] "Generic (PLEG): container finished" podID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerID="b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31" exitCode=0
Nov 03 12:36:08.492932 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:08.492701    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" event={"ID":"5dae7f1a-c57a-4707-8f78-03a0464be8f6","Type":"ContainerDied","Data":"b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"}
Nov 03 12:36:08.493243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:08.493222    4546 scope.go:122] "RemoveContainer" containerID="b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"
Nov 03 12:36:08.495041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:08.495014    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:36:08.495157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:08.495088    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:36:08.495157 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:08.495125    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:36:09.501135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:09.501100    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" event={"ID":"5dae7f1a-c57a-4707-8f78-03a0464be8f6","Type":"ContainerStarted","Data":"4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372"}
Nov 03 12:36:10.404458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:10.404428    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:36:10.404697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:10.404675    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 03 12:36:10.404780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:10.404725    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 03 12:36:11.403697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:11.403662    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 03 12:36:11.404154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:11.403714    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 03 12:36:12.403645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:12.403606    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 03 12:36:12.403867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:12.403669    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 03 12:36:13.404126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:13.404082    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 03 12:36:13.404581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:13.404150    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 03 12:36:14.404427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:14.404395    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 03 12:36:14.404931 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:14.404457    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 03 12:36:15.409267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:15.409234    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:36:15.409730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:15.409706    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:36:15.410507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:15.410487    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 12:36:16.044963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:16.044900    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"]
Nov 03 12:36:16.048630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:16.048603    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58866b73-2rjtm"]
Nov 03 12:36:16.573492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:16.573460    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162c39ae-9d38-4fb7-8599-dd6d897db1f7" path="/var/lib/kubelet/pods/162c39ae-9d38-4fb7-8599-dd6d897db1f7/volumes"
Nov 03 12:36:17.055842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:17.055808    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"]
Nov 03 12:36:17.096289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:17.096244    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955e1cab-2xkcp"]
Nov 03 12:36:18.573889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:18.573861    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5515370d-003a-4f7e-98b1-beb01bf18842" path="/var/lib/kubelet/pods/5515370d-003a-4f7e-98b1-beb01bf18842/volumes"
Nov 03 12:36:19.046287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:19.046251    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"]
Nov 03 12:36:19.092305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:19.092270    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ceb6b1-vp9qb"]
Nov 03 12:36:20.573905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:20.573873    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c81586-3b88-418d-a3f5-742caf20efe4" path="/var/lib/kubelet/pods/84c81586-3b88-418d-a3f5-742caf20efe4/volumes"
Nov 03 12:36:22.044263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:22.044234    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"]
Nov 03 12:36:22.089544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:22.089517    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1b9bcb-92r7q"]
Nov 03 12:36:22.574884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:22.574854    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b9efbf-3414-4bdc-b7de-bc53af0e1963" path="/var/lib/kubelet/pods/37b9efbf-3414-4bdc-b7de-bc53af0e1963/volumes"
Nov 03 12:36:26.046307 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:26.046269    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"]
Nov 03 12:36:26.091528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:26.091500    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9ede3e-tst6m"]
Nov 03 12:36:26.573804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:26.573775    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2decc78-a8ab-4363-94a0-b5270d18d3e8" path="/var/lib/kubelet/pods/b2decc78-a8ab-4363-94a0-b5270d18d3e8/volumes"
Nov 03 12:36:32.044275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:32.044234    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"]
Nov 03 12:36:32.089969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:32.089936    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab008ca-sm4f5"]
Nov 03 12:36:32.575021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:32.574983    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3db6af-6c64-43bd-99cf-f26d937313b8" path="/var/lib/kubelet/pods/ce3db6af-6c64-43bd-99cf-f26d937313b8/volumes"
Nov 03 12:36:35.731315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:35.731285    4546 generic.go:356] "Generic (PLEG): container finished" podID="defce8d6-9532-451e-8ffc-7335e035cf83" containerID="24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57" exitCode=0
Nov 03 12:36:35.731777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:35.731353    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" event={"ID":"defce8d6-9532-451e-8ffc-7335e035cf83","Type":"ContainerDied","Data":"24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"}
Nov 03 12:36:35.731777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:35.731719    4546 scope.go:122] "RemoveContainer" containerID="24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"
Nov 03 12:36:35.733264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:35.733240    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:36:35.733329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:35.733305    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:36:35.733373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:35.733340    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:36:36.741482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:36.741442    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" event={"ID":"defce8d6-9532-451e-8ffc-7335e035cf83","Type":"ContainerStarted","Data":"563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31"}
Nov 03 12:36:37.531564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:37.531530    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:36:37.533526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:37.533492    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 12:36:37.533699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:37.533555    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 12:36:37.660866 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:36:37.660782    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:36:38.532547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:38.532410    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 12:36:38.532547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:38.532467    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 12:36:39.531321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:39.531281    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 12:36:39.531729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:39.531690    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 12:36:40.530732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:40.530696    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 12:36:40.531353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:40.531324    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 12:36:40.779889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:40.779852    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:36:41.532198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:41.532157    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 12:36:41.532623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:41.532233    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 12:36:42.050691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:42.050659    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"]
Nov 03 12:36:42.096776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:42.096732    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b198df22-66hwq"]
Nov 03 12:36:42.535906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:42.535875    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:36:42.537090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:42.537069    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:36:42.574992 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:42.574967    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267fdba9-1d5d-41ce-9abd-c306d457358c" path="/var/lib/kubelet/pods/267fdba9-1d5d-41ce-9abd-c306d457358c/volumes"
Nov 03 12:36:55.092356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:55.092321    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"]
Nov 03 12:36:55.094722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:55.094698    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6306015d-rfx2z"]
Nov 03 12:36:56.573838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:36:56.573809    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4af4b40-a245-4146-b652-dd9333b0a4eb" path="/var/lib/kubelet/pods/a4af4b40-a245-4146-b652-dd9333b0a4eb/volumes"
Nov 03 12:37:00.872682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:00.872654    4546 scope.go:122] "RemoveContainer" containerID="e8b12764b28d503791e3d2a11ea43649239275e01fbb32d69c09834de96c4b3b"
Nov 03 12:37:00.883584 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:00.883554    4546 scope.go:122] "RemoveContainer" containerID="23539fc5760ad19a4f777a8704050895896730a68551067e2fafc498b59d8c6e"
Nov 03 12:37:00.892919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:00.892892    4546 scope.go:122] "RemoveContainer" containerID="c4d57b6a6219c4dc835043a2f4668dfd81db782e09eb8e4a45cd4b386b6bd1b6"
Nov 03 12:37:00.902065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:00.902045    4546 scope.go:122] "RemoveContainer" containerID="6a2422202fab60de2013cb9947d993b6acc67c02fa7a5239047d66f575358754"
Nov 03 12:37:03.047085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:03.047058    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"]
Nov 03 12:37:03.093366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:03.093338    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"]
Nov 03 12:37:03.096351 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:03.096335    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad5d84d-8ghrl"]
Nov 03 12:37:03.098906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:03.098881    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33bf2953-lgsgm"]
Nov 03 12:37:04.573986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:04.573954    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21358f3-7294-4c0c-a074-1175b9cca481" path="/var/lib/kubelet/pods/a21358f3-7294-4c0c-a074-1175b9cca481/volumes"
Nov 03 12:37:04.574451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:04.574435    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d860e474-de0b-4aa8-97af-34315272aaa2" path="/var/lib/kubelet/pods/d860e474-de0b-4aa8-97af-34315272aaa2/volumes"
Nov 03 12:37:07.652541 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:37:07.652505    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:37:10.043284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:10.043248    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"]
Nov 03 12:37:10.088841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:10.088815    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013ab3ec-zzqjk"]
Nov 03 12:37:10.573292 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:10.573261    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6069fd59-2d21-4fcb-bf1a-deb14a1526d3" path="/var/lib/kubelet/pods/6069fd59-2d21-4fcb-bf1a-deb14a1526d3/volumes"
Nov 03 12:37:19.042615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:19.042581    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"]
Nov 03 12:37:19.089490 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:19.089454    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32bc120-fb4ps"]
Nov 03 12:37:20.087139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:20.087099    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"]
Nov 03 12:37:20.090246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:20.090218    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616c7952-tjngl"]
Nov 03 12:37:20.574314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:20.574280    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c89775-6b94-4786-8e96-da6bcb0a3530" path="/var/lib/kubelet/pods/24c89775-6b94-4786-8e96-da6bcb0a3530/volumes"
Nov 03 12:37:20.574780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:20.574766    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f1d44e-b23f-430b-a484-bba1cda31a06" path="/var/lib/kubelet/pods/b3f1d44e-b23f-430b-a484-bba1cda31a06/volumes"
Nov 03 12:37:24.614467 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.614432    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"]
Nov 03 12:37:24.615359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615333    4546 cpu_manager.go:414] "containerMap: removing container" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy"
Nov 03 12:37:24.615359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615358    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy"
Nov 03 12:37:24.615551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615389    4546 cpu_manager.go:414] "containerMap: removing container" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy"
Nov 03 12:37:24.615551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615398    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy"
Nov 03 12:37:24.615551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615418    4546 cpu_manager.go:414] "containerMap: removing container" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy"
Nov 03 12:37:24.615551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615426    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy"
Nov 03 12:37:24.615551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615469    4546 cpu_manager.go:414] "containerMap: removing container" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy"
Nov 03 12:37:24.615551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615478    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy"
Nov 03 12:37:24.615923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615628    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a45b90-0b75-4a20-837a-8f364ea4cd28" containerName="istio-proxy"
Nov 03 12:37:24.615923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615646    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="420474ed-cdc6-4c0e-bd58-53dd11cfa678" containerName="istio-proxy"
Nov 03 12:37:24.615923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615658    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy"
Nov 03 12:37:24.615923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.615668    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a0349b-1f17-4df2-9167-6acd005107d1" containerName="istio-proxy"
Nov 03 12:37:24.619439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.619422    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.673122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.673088    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"]
Nov 03 12:37:24.713889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.713849    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ab27b8\"/\"default-dockercfg-6s92r\""
Nov 03 12:37:24.714074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.713956    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ab27b8\"/\"openshift-service-ca.crt\""
Nov 03 12:37:24.714074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.713982    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ab27b8\"/\"auth-test-model-ab27b8-kserve-self-signed-certs\""
Nov 03 12:37:24.714473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.714449    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ab27b8\"/\"kube-root-ca.crt\""
Nov 03 12:37:24.760272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.760235    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.761553 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.761516    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.761851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.761829    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dprv\" (UniqueName: \"kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.762222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.762203    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.762337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.762300    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.762437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.762351    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873281    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873355    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873395    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873434    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873457    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873575    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dprv\" (UniqueName: \"kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873807    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.873899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.873815    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.874200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.874160    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.876429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.876410    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.876728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.876707    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.929159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.929127    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dprv\" (UniqueName: \"kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv\") pod \"auth-test-model-ab27b8-kserve-7459dc6469-5nzhc\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") " pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:24.933088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:24.933064    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:25.301073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:25.301044    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"]
Nov 03 12:37:25.301721 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:37:25.301698    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6141c84b_9c1f_41be_a999_835d394c5c5b.slice/crio-47e92ebe755d2c6dba37213ebbb278e1e95dd19e1eecdd66079d228d2141814b WatchSource:0}: Error finding container 47e92ebe755d2c6dba37213ebbb278e1e95dd19e1eecdd66079d228d2141814b: Status 404 returned error can't find the container with id 47e92ebe755d2c6dba37213ebbb278e1e95dd19e1eecdd66079d228d2141814b
Nov 03 12:37:26.119424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:26.119381    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc" event={"ID":"6141c84b-9c1f-41be-a999-835d394c5c5b","Type":"ContainerStarted","Data":"bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2"}
Nov 03 12:37:26.119424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:26.119429    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc" event={"ID":"6141c84b-9c1f-41be-a999-835d394c5c5b","Type":"ContainerStarted","Data":"47e92ebe755d2c6dba37213ebbb278e1e95dd19e1eecdd66079d228d2141814b"}
Nov 03 12:37:26.226406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:26.226351    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc" podStartSLOduration=2.226334669 podStartE2EDuration="2.226334669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:37:26.225426949 +0000 UTC m=+2787.797010880" watchObservedRunningTime="2026-11-03 12:37:26.226334669 +0000 UTC m=+2787.797918615"
Nov 03 12:37:26.933373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:26.933336    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:34.933673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:34.933633    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:34.947378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:34.947343    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:35.090976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.090931    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"]
Nov 03 12:37:35.095711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.095682    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.144805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.144773    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"kube-root-ca.crt\""
Nov 03 12:37:35.145074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.144816    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"openshift-service-ca.crt\""
Nov 03 12:37:35.145280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.145261    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"default-dockercfg-2gjqq\""
Nov 03 12:37:35.145519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.145485    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"auth-test-model-cacb84-kserve-self-signed-certs\""
Nov 03 12:37:35.153926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.153876    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"]
Nov 03 12:37:35.217801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.217736    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:37:35.253045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.252994    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.253288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.253113    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7rn\" (UniqueName: \"kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.253288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.253147    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.253288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.253195    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.253461 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.253332    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.253551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.253528    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.354912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.354873    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.354982    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7rn\" (UniqueName: \"kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355018    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355067    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355217    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355316 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355279    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355348    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355730    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.355924 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.355835    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.358148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.358117    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.358303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.358279    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.408529 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.408499    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7rn\" (UniqueName: \"kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn\") pod \"auth-test-model-cacb84-kserve-74554d4645-kmhxh\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.410407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.410379    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:35.788009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:35.787963    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"]
Nov 03 12:37:35.789190 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:37:35.789160    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45d4b012_69a9_4698_8c37_b70d5c0e6296.slice/crio-f4cb6e51bf14ee1e95a27f6f0fc2bbd3589e7ebd69c9572a922776e56a0a5b82 WatchSource:0}: Error finding container f4cb6e51bf14ee1e95a27f6f0fc2bbd3589e7ebd69c9572a922776e56a0a5b82: Status 404 returned error can't find the container with id f4cb6e51bf14ee1e95a27f6f0fc2bbd3589e7ebd69c9572a922776e56a0a5b82
Nov 03 12:37:36.216227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:36.216188    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh" event={"ID":"45d4b012-69a9-4698-8c37-b70d5c0e6296","Type":"ContainerStarted","Data":"e605645282ee08507ce5d076e7d788a8e6ff76a6127ce0842b93ed63841fd2ab"}
Nov 03 12:37:36.217097 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:36.216245    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh" event={"ID":"45d4b012-69a9-4698-8c37-b70d5c0e6296","Type":"ContainerStarted","Data":"f4cb6e51bf14ee1e95a27f6f0fc2bbd3589e7ebd69c9572a922776e56a0a5b82"}
Nov 03 12:37:36.323380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:36.323324    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh" podStartSLOduration=1.323306911 podStartE2EDuration="1.323306911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:37:36.321264596 +0000 UTC m=+2797.892848528" watchObservedRunningTime="2026-11-03 12:37:36.323306911 +0000 UTC m=+2797.894890822"
Nov 03 12:37:36.410813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:36.410775    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:37.042366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:37.042329    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"]
Nov 03 12:37:37.088819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:37.088785    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e41ba69-m7lrh"]
Nov 03 12:37:37.658375 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:37:37.658339    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:37:38.586691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:38.586651    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f90670-2e44-4225-adbd-356d5fa485b7" path="/var/lib/kubelet/pods/38f90670-2e44-4225-adbd-356d5fa485b7/volumes"
Nov 03 12:37:45.411272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:45.411230    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:45.424029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:45.423997    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:37:46.304993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:37:46.304955    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:38:00.947826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:00.947792    4546 scope.go:122] "RemoveContainer" containerID="324ec87d3ed4ae368af9dd13d788506f77b5957bb3f2260dece85bf356cc7a78"
Nov 03 12:38:00.958235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:00.958217    4546 scope.go:122] "RemoveContainer" containerID="b6ceba5c65debeece812131e20ac5b8980bcf985cc625dac875dbba96c04d6c4"
Nov 03 12:38:00.967697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:00.967673    4546 scope.go:122] "RemoveContainer" containerID="0fe173731c36289a78fc5da3c7f41c5430d38cdf2a7f7155a4b39f9f7fc17b4c"
Nov 03 12:38:00.977330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:00.977304    4546 scope.go:122] "RemoveContainer" containerID="c479b3447a26b01e512640a2a903f2c8f32f0a1943e0317db6024242ee19ce0a"
Nov 03 12:38:00.988393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:00.988374    4546 scope.go:122] "RemoveContainer" containerID="60826e700f00a0097f7fcb56586e0379b37d4c1e24a9c8986e00f1bf56c28ec2"
Nov 03 12:38:00.998000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:00.997976    4546 scope.go:122] "RemoveContainer" containerID="a0ac8368da9017b310f4f0f657fbed47ae18656c43c92d44e251a70920959a1f"
Nov 03 12:38:01.007657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:01.007634    4546 scope.go:122] "RemoveContainer" containerID="e9647365c7734853705b3f3c218783447f4e7cd44d79d4e41574c636258e746e"
Nov 03 12:38:01.017056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:01.017033    4546 scope.go:122] "RemoveContainer" containerID="87f64dd4e08c320b389ec94f6498fc4b1e1e4fccdced0c41be0697016d1f5583"
Nov 03 12:38:01.026982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:01.026955    4546 scope.go:122] "RemoveContainer" containerID="25d8a7d36a575ae26c3ec4b01054cbe45cafaad1df6a11000259b5fbad6c35f9"
Nov 03 12:38:01.036844 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:01.036822    4546 scope.go:122] "RemoveContainer" containerID="2561cab92d7e95d403aa2d187b0449ba12b7f9ba5cb554b8bafbc6bf96444216"
Nov 03 12:38:03.254991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:03.254957    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"]
Nov 03 12:38:03.255616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:03.255231    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy" containerID="cri-o://43746d6d7530aa9c3deb2b9a1176d8d796785374efa48119774743cb4f6934ef" gracePeriod=30
Nov 03 12:38:07.653135 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:38:07.653040    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:38:08.523410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.523314    4546 generic.go:356] "Generic (PLEG): container finished" podID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerID="43746d6d7530aa9c3deb2b9a1176d8d796785374efa48119774743cb4f6934ef" exitCode=0
Nov 03 12:38:08.523410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.523392    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" event={"ID":"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4","Type":"ContainerDied","Data":"43746d6d7530aa9c3deb2b9a1176d8d796785374efa48119774743cb4f6934ef"}
Nov 03 12:38:08.523588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.523459    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj" event={"ID":"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4","Type":"ContainerDied","Data":"f1478dd78bd46e05e66221c1164f6cb26acee22a902697df75d7a491644cb28f"}
Nov 03 12:38:08.523588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.523477    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1478dd78bd46e05e66221c1164f6cb26acee22a902697df75d7a491644cb28f"
Nov 03 12:38:08.536974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.536949    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:38:08.551108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551084    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551167    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551193    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551247    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551298    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551393    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551427    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551486    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.551533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551492    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:08.551639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551602    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:08.551988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551772    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:08.552085 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.551993    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:08.552215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.552196    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.552331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.552217    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.552331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.552244    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.552331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.552258    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.554016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.553987    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "kube-api-access-qd65t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:08.554117 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.554094    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:08.554161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.554115    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:08.554286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.554264    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:08.653762 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.653715    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data\") pod \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\" (UID: \"b010bc1e-4478-431a-adb7-fb4f5e9ba9a4\") "
Nov 03 12:38:08.654194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.653989    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data" pod "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" (UID: "b010bc1e-4478-431a-adb7-fb4f5e9ba9a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:08.654357 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.654338    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.654405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.654361    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.654405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.654370    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd65t\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-kube-api-access-qd65t\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.654405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.654379    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:08.654405 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:08.654388    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:09.530070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:09.530037    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"
Nov 03 12:38:09.729967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:09.729931    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"]
Nov 03 12:38:09.775152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:09.775117    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-dnwvj"]
Nov 03 12:38:10.575302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:10.575271    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" path="/var/lib/kubelet/pods/b010bc1e-4478-431a-adb7-fb4f5e9ba9a4/volumes"
Nov 03 12:38:18.367617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:18.367580    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"]
Nov 03 12:38:18.368177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:18.367841    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy" containerID="cri-o://4d8d2786750ce14071cf05ded403bc7d81863e4f2a82cc4fb8a128527c06a1a0" gracePeriod=30
Nov 03 12:38:18.414860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:18.414810    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"]
Nov 03 12:38:18.415089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:18.415052    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy" containerID="cri-o://55aef8c02249789c450391f6ae3781d26358c64ec6dd465e77ff479f60718bc7" gracePeriod=30
Nov 03 12:38:23.636906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.636874    4546 generic.go:356] "Generic (PLEG): container finished" podID="a9391c99-e893-4007-9dac-6ac0f722953f" containerID="4d8d2786750ce14071cf05ded403bc7d81863e4f2a82cc4fb8a128527c06a1a0" exitCode=0
Nov 03 12:38:23.637382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.636950    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" event={"ID":"a9391c99-e893-4007-9dac-6ac0f722953f","Type":"ContainerDied","Data":"4d8d2786750ce14071cf05ded403bc7d81863e4f2a82cc4fb8a128527c06a1a0"}
Nov 03 12:38:23.638986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.638924    4546 generic.go:356] "Generic (PLEG): container finished" podID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerID="55aef8c02249789c450391f6ae3781d26358c64ec6dd465e77ff479f60718bc7" exitCode=0
Nov 03 12:38:23.639107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.638999    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" event={"ID":"cbc0e2b2-b5ad-4826-9753-a8da70d28a10","Type":"ContainerDied","Data":"55aef8c02249789c450391f6ae3781d26358c64ec6dd465e77ff479f60718bc7"}
Nov 03 12:38:23.694095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.694063    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:38:23.738619 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.738594    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:38:23.788162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788131    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788175    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788196    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788245    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788385    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788570 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788468    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.788627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788576    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788610    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788645    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788673    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:23.788909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788813    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.788909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788849    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo\") pod \"a9391c99-e893-4007-9dac-6ac0f722953f\" (UID: \"a9391c99-e893-4007-9dac-6ac0f722953f\") "
Nov 03 12:38:23.788909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.788864    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.789139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.789084    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.789728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.789705    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.789880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.789733    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.789880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.789768    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.789880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.789784    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9391c99-e893-4007-9dac-6ac0f722953f-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.789880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.789799    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.790952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.790922    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.790952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.790933    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:23.791109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.790971    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "kube-api-access-lhlgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:23.791109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.791088    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo" pod "a9391c99-e893-4007-9dac-6ac0f722953f" (UID: "a9391c99-e893-4007-9dac-6ac0f722953f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:23.890621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890526    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.890621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890585    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.890910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890696    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.890910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890784    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.890910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890856    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.891087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890912    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.891087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890967    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.891087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.890995    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.891087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.891033    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.891087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.891042    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.891345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.891128    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert\") pod \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\" (UID: \"cbc0e2b2-b5ad-4826-9753-a8da70d28a10\") "
Nov 03 12:38:23.891345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.891259    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.891627 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.891383    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.891688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.891638    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:23.892123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892097    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892128    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9391c99-e893-4007-9dac-6ac0f722953f-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892144    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892159    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892176    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhlgj\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-kube-api-access-lhlgj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892191    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892206    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9391c99-e893-4007-9dac-6ac0f722953f-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892222    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9391c99-e893-4007-9dac-6ac0f722953f-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.892262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.892236    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.893304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.893276    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:23.893404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.893366    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "kube-api-access-s4p2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:23.894031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.894010    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:23.894170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.894154    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy" pod "cbc0e2b2-b5ad-4826-9753-a8da70d28a10" (UID: "cbc0e2b2-b5ad-4826-9753-a8da70d28a10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:23.993895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.993848    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.993895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.993894    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.994180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.993911    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4p2h\" (UniqueName: \"kubernetes.io/projected/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-kube-api-access-s4p2h\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.994180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:23.993927    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc0e2b2-b5ad-4826-9753-a8da70d28a10-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:24.647829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.647803    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"
Nov 03 12:38:24.648345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.647803    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk" event={"ID":"a9391c99-e893-4007-9dac-6ac0f722953f","Type":"ContainerDied","Data":"f754a8477ae668f19b894487f26bdf37ad6060ab9b809ed5c235f07171ca9d59"}
Nov 03 12:38:24.648345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.647950    4546 scope.go:122] "RemoveContainer" containerID="4d8d2786750ce14071cf05ded403bc7d81863e4f2a82cc4fb8a128527c06a1a0"
Nov 03 12:38:24.649802 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.649773    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt" event={"ID":"cbc0e2b2-b5ad-4826-9753-a8da70d28a10","Type":"ContainerDied","Data":"dde0be3d13da91b12c72259fe3855f01e2dde680a42aaf837bf4c8c478794c56"}
Nov 03 12:38:24.649908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.649805    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"
Nov 03 12:38:24.658851 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.658831    4546 scope.go:122] "RemoveContainer" containerID="55aef8c02249789c450391f6ae3781d26358c64ec6dd465e77ff479f60718bc7"
Nov 03 12:38:24.803550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.803507    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"]
Nov 03 12:38:24.850609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.850570    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5cvqk"]
Nov 03 12:38:24.950504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.950418    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"]
Nov 03 12:38:24.955168 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:24.955139    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-8hbmt"]
Nov 03 12:38:26.575302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:26.575270    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" path="/var/lib/kubelet/pods/a9391c99-e893-4007-9dac-6ac0f722953f/volumes"
Nov 03 12:38:26.575815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:26.575800    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" path="/var/lib/kubelet/pods/cbc0e2b2-b5ad-4826-9753-a8da70d28a10/volumes"
Nov 03 12:38:29.773416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.773371    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"]
Nov 03 12:38:29.775304 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.775273    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy"
Nov 03 12:38:29.775492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.775478    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy"
Nov 03 12:38:29.775702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.775689    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy"
Nov 03 12:38:29.775813 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.775801    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy"
Nov 03 12:38:29.775923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.775913    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy"
Nov 03 12:38:29.775999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.775990    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy"
Nov 03 12:38:29.776369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.776353    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b010bc1e-4478-431a-adb7-fb4f5e9ba9a4" containerName="istio-proxy"
Nov 03 12:38:29.776478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.776469    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9391c99-e893-4007-9dac-6ac0f722953f" containerName="istio-proxy"
Nov 03 12:38:29.776567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.776558    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc0e2b2-b5ad-4826-9753-a8da70d28a10" containerName="istio-proxy"
Nov 03 12:38:29.781483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.781459    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.870849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.870804    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.870849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.870855    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.871125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.870955    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.871125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.871001    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5t8\" (UniqueName: \"kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.871251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.871148    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.871251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.871183    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.882142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.882117    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"kube-root-ca.crt\""
Nov 03 12:38:29.882319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.882304    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"auth-test-model-8570a0-kserve-self-signed-certs\""
Nov 03 12:38:29.882385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.882369    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"openshift-service-ca.crt\""
Nov 03 12:38:29.885906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.885888    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"]
Nov 03 12:38:29.886677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.886664    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"default-dockercfg-rrntn\""
Nov 03 12:38:29.973651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.973602    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.973651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.973647    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5t8\" (UniqueName: \"kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.973966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.973772    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.973966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.973818    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.974074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.974013    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.974122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.974097    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.974180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.974133    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.974593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.974563    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.974781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.974691    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.977455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.977434    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:29.977717 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:29.977699    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:30.027796 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.027699    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5t8\" (UniqueName: \"kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8\") pod \"auth-test-model-8570a0-kserve-6784ff67c7-tpv9m\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:30.093347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.093316    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:30.461172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.461139    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"]
Nov 03 12:38:30.462019 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:38:30.461992    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfadae778_5912_4c36_a576_1d31e79af6d7.slice/crio-3ba150c9449d160196bf4632fccbeae296bff559479ad61062e82b66237dff7b WatchSource:0}: Error finding container 3ba150c9449d160196bf4632fccbeae296bff559479ad61062e82b66237dff7b: Status 404 returned error can't find the container with id 3ba150c9449d160196bf4632fccbeae296bff559479ad61062e82b66237dff7b
Nov 03 12:38:30.694160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.694117    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m" event={"ID":"fadae778-5912-4c36-a576-1d31e79af6d7","Type":"ContainerStarted","Data":"24e3a84915c00b9f375786d540c4cb5552ef69c51fea440345467e8562b45580"}
Nov 03 12:38:30.694346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.694172    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m" event={"ID":"fadae778-5912-4c36-a576-1d31e79af6d7","Type":"ContainerStarted","Data":"3ba150c9449d160196bf4632fccbeae296bff559479ad61062e82b66237dff7b"}
Nov 03 12:38:30.800004 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.799948    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m" podStartSLOduration=1.799933403 podStartE2EDuration="1.799933403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:38:30.799078598 +0000 UTC m=+2852.370662509" watchObservedRunningTime="2026-11-03 12:38:30.799933403 +0000 UTC m=+2852.371517314"
Nov 03 12:38:30.972262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.972226    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"]
Nov 03 12:38:30.972524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.972500    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy" containerID="cri-o://639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb" gracePeriod=30
Nov 03 12:38:30.975675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.975648    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"]
Nov 03 12:38:30.975930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:30.975902    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy" containerID="cri-o://65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7" gracePeriod=30
Nov 03 12:38:31.093965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:31.093891    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:36.269646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.269617    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:38:36.294165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.294135    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:38:36.370328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370293    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370385    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.370545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370421    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370445    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370505    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.370545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370523    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370559    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370596    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370640    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370668    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370678    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370726    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370828    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370841    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.370877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370866    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370919    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370939    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370994    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo\") pod \"17cf3193-e29c-4cf1-9155-624b3c55af65\" (UID: \"17cf3193-e29c-4cf1-9155-624b3c55af65\") "
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371006    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.370947    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371067    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371104    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371162    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket\") pod \"5bca048c-17d5-4693-87cf-df55c4d16ddc\" (UID: \"5bca048c-17d5-4693-87cf-df55c4d16ddc\") "
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371242    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371267    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:36.371518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371295    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371624    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371973    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.371997    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bca048c-17d5-4693-87cf-df55c4d16ddc-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372014    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372028    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372042    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372055    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372069    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372084    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372100    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf3193-e29c-4cf1-9155-624b3c55af65-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.372694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.372174    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.373258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.373231    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:36.373535 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.373508    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:36.373870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.373846    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.374122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.374102    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:36.374513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.374489    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:36.374596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.374523    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl" pod "17cf3193-e29c-4cf1-9155-624b3c55af65" (UID: "17cf3193-e29c-4cf1-9155-624b3c55af65"). InnerVolumeSpecName "kube-api-access-mglwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:36.374661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.374620    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "kube-api-access-pmtlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:36.375164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.375136    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy" pod "5bca048c-17d5-4693-87cf-df55c4d16ddc" (UID: "5bca048c-17d5-4693-87cf-df55c4d16ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:36.473587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473549    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473584    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473594    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473604    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglwl\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-kube-api-access-mglwl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473616    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473624    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473634    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtlp\" (UniqueName: \"kubernetes.io/projected/5bca048c-17d5-4693-87cf-df55c4d16ddc-kube-api-access-pmtlp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473646    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf3193-e29c-4cf1-9155-624b3c55af65-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.473856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.473660    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bca048c-17d5-4693-87cf-df55c4d16ddc-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:36.748371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.748326    4546 generic.go:356] "Generic (PLEG): container finished" podID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerID="639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb" exitCode=0
Nov 03 12:38:36.748661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.748642    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" event={"ID":"17cf3193-e29c-4cf1-9155-624b3c55af65","Type":"ContainerDied","Data":"639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb"}
Nov 03 12:38:36.748788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.748724    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"
Nov 03 12:38:36.748788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.748768    4546 scope.go:122] "RemoveContainer" containerID="639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb"
Nov 03 12:38:36.748901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.748733    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp" event={"ID":"17cf3193-e29c-4cf1-9155-624b3c55af65","Type":"ContainerDied","Data":"daacffe4bb97356d7cfbb813ca7afe74a864f513a9e472ab8a2e1e3a041c0b82"}
Nov 03 12:38:36.761245 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.761208    4546 generic.go:356] "Generic (PLEG): container finished" podID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerID="65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7" exitCode=0
Nov 03 12:38:36.761425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.761278    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"
Nov 03 12:38:36.761425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.761294    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" event={"ID":"5bca048c-17d5-4693-87cf-df55c4d16ddc","Type":"ContainerDied","Data":"65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7"}
Nov 03 12:38:36.761425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.761355    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg" event={"ID":"5bca048c-17d5-4693-87cf-df55c4d16ddc","Type":"ContainerDied","Data":"71016a8917d2b4264738e09955d36547384411871dde8c862dbb8edda9908c23"}
Nov 03 12:38:36.763571 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.763550    4546 scope.go:122] "RemoveContainer" containerID="639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb"
Nov 03 12:38:36.763909 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:38:36.763880    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb\": container with ID starting with 639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb not found: ID does not exist" containerID="639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb"
Nov 03 12:38:36.764002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.763920    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb"} err="failed to get container status \"639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb\": rpc error: code = NotFound desc = could not find container \"639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb\": container with ID starting with 639e28bdb62ba8e7ad41fef310a2fa22823b6533f186b37b0e00fa6eae05e6bb not found: ID does not exist"
Nov 03 12:38:36.764002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.763944    4546 scope.go:122] "RemoveContainer" containerID="65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7"
Nov 03 12:38:36.775223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.775202    4546 scope.go:122] "RemoveContainer" containerID="65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7"
Nov 03 12:38:36.775554 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:38:36.775525    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7\": container with ID starting with 65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7 not found: ID does not exist" containerID="65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7"
Nov 03 12:38:36.775677 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.775554    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7"} err="failed to get container status \"65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7\": rpc error: code = NotFound desc = could not find container \"65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7\": container with ID starting with 65f658c9256e872d6082788eb5edbaea52998aa8835f6034b2da242bd54126e7 not found: ID does not exist"
Nov 03 12:38:36.903840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.903702    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"]
Nov 03 12:38:36.907174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:36.907143    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-w58vp"]
Nov 03 12:38:37.008483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:37.008451    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"]
Nov 03 12:38:37.012218 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:37.012185    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jn4tg"]
Nov 03 12:38:37.652099 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:38:37.652059    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:38:38.574970 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:38.574932    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" path="/var/lib/kubelet/pods/17cf3193-e29c-4cf1-9155-624b3c55af65/volumes"
Nov 03 12:38:38.575566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:38.575542    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" path="/var/lib/kubelet/pods/5bca048c-17d5-4693-87cf-df55c4d16ddc/volumes"
Nov 03 12:38:40.093997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:40.093959    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:40.106707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:40.106680    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:40.799232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:40.799203    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:38:45.001887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.001852    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"]
Nov 03 12:38:45.002440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.002423    4546 cpu_manager.go:414] "containerMap: removing container" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy"
Nov 03 12:38:45.002440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.002439    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy"
Nov 03 12:38:45.002637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.002470    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy"
Nov 03 12:38:45.002637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.002477    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy"
Nov 03 12:38:45.002637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.002594    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cf3193-e29c-4cf1-9155-624b3c55af65" containerName="istio-proxy"
Nov 03 12:38:45.002637 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.002609    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bca048c-17d5-4693-87cf-df55c4d16ddc" containerName="istio-proxy"
Nov 03 12:38:45.007590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.007569    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.073193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.073161    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-743d9b\"/\"openshift-service-ca.crt\""
Nov 03 12:38:45.102912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.102883    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-743d9b\"/\"kube-root-ca.crt\""
Nov 03 12:38:45.103150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.103129    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-743d9b\"/\"auth-test-model-743d9b-kserve-self-signed-certs\""
Nov 03 12:38:45.103241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.103129    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-743d9b\"/\"default-dockercfg-49tx7\""
Nov 03 12:38:45.106373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.106346    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"]
Nov 03 12:38:45.188015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.187975    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.188225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.188124    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.188225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.188162    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.188336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.188317    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.188382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.188357    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.188495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.188473    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.289457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.289346    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.289647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.289467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.289647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.289505    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.289647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.289600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.289647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.289642    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.289879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.289710    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.290042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.290014    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.290125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.290059    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.290184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.290131    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.292000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.291976    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.292273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.292255    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.343114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.343074    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc\") pod \"auth-test-model-743d9b-kserve-5599f6b469-clb5j\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") " pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:45.622006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:45.621909    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:46.001070 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:46.001040    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"]
Nov 03 12:38:46.001946 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:38:46.001914    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf382114d_4d9e_44bd_aff6_e6d651c11803.slice/crio-f063f3d12cdb3ccb7397db4331efdb21d1db60b2fa5e8721f8e6aa89fde81152 WatchSource:0}: Error finding container f063f3d12cdb3ccb7397db4331efdb21d1db60b2fa5e8721f8e6aa89fde81152: Status 404 returned error can't find the container with id f063f3d12cdb3ccb7397db4331efdb21d1db60b2fa5e8721f8e6aa89fde81152
Nov 03 12:38:46.830064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:46.830024    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j" event={"ID":"f382114d-4d9e-44bd-aff6-e6d651c11803","Type":"ContainerStarted","Data":"ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52"}
Nov 03 12:38:46.830265 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:46.830076    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j" event={"ID":"f382114d-4d9e-44bd-aff6-e6d651c11803","Type":"ContainerStarted","Data":"f063f3d12cdb3ccb7397db4331efdb21d1db60b2fa5e8721f8e6aa89fde81152"}
Nov 03 12:38:46.938119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:46.938060    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j" podStartSLOduration=2.938042944 podStartE2EDuration="2.938042944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:38:46.935774067 +0000 UTC m=+2868.507357976" watchObservedRunningTime="2026-11-03 12:38:46.938042944 +0000 UTC m=+2868.509626855"
Nov 03 12:38:47.628525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:47.628482    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:55.628150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:55.628112    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:55.641391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:55.641366    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:38:55.901194 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:38:55.901118    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:39:01.109172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:01.109139    4546 scope.go:122] "RemoveContainer" containerID="7c73ac070eb6bbf36662e796c13612256fb06035ba152778adc49c6957af812b"
Nov 03 12:39:01.118819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:01.118796    4546 scope.go:122] "RemoveContainer" containerID="0cdbaa9ab8efe9057ac98e7f10fd3a96eb1e8b32d398b93cc40d0a0ba3c58b57"
Nov 03 12:39:01.129710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:01.129688    4546 scope.go:122] "RemoveContainer" containerID="98cd3713aad3b49b5be59df1b27cda9036051b59f7f07d8ea6bf595307e5f14a"
Nov 03 12:39:01.138582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:01.138565    4546 scope.go:122] "RemoveContainer" containerID="670656fc8f6827c9063c7f09d10b3bb1d42d52eb2c486ac9f7da306d510b4ae5"
Nov 03 12:39:01.147146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:01.147126    4546 scope.go:122] "RemoveContainer" containerID="43746d6d7530aa9c3deb2b9a1176d8d796785374efa48119774743cb4f6934ef"
Nov 03 12:39:07.655535 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:39:07.655492    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:39:37.652608 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:39:37.652507    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:39:44.588329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:44.588298    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"]
Nov 03 12:39:44.588873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:44.588476    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy" containerID="cri-o://a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081" gracePeriod=30
Nov 03 12:39:49.854687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:49.854620    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:39:50.019794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.019744    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.019974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.019872    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.019974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.019903    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.019974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.019938    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.020162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.019982    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:50.020162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.019989    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.020162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020039    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.020162 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020153    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.020388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020187    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.020388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020189    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:50.020388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020219    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt\") pod \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\" (UID: \"cf06f6d5-420f-4f6a-9103-ccff5cd0b392\") "
Nov 03 12:39:50.020984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020960    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.021107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.020990    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.021305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.021285    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:50.021595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.021561    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:50.021900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.021842    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:39:50.023024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.022726    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:39:50.024144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.024122    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:50.024144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.024126    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "kube-api-access-xwvxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:39:50.024389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.024369    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo" pod "cf06f6d5-420f-4f6a-9103-ccff5cd0b392" (UID: "cf06f6d5-420f-4f6a-9103-ccff5cd0b392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:39:50.122648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122567    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.122648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122597    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.122648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122610    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.122648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122621    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.122648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122634    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.122648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122651    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.123012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.122664    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwvxt\" (UniqueName: \"kubernetes.io/projected/cf06f6d5-420f-4f6a-9103-ccff5cd0b392-kube-api-access-xwvxt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:50.250684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.250641    4546 generic.go:356] "Generic (PLEG): container finished" podID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerID="a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081" exitCode=0
Nov 03 12:39:50.250891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.250743    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" event={"ID":"cf06f6d5-420f-4f6a-9103-ccff5cd0b392","Type":"ContainerDied","Data":"a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081"}
Nov 03 12:39:50.250891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.250843    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2" event={"ID":"cf06f6d5-420f-4f6a-9103-ccff5cd0b392","Type":"ContainerDied","Data":"0e9dd90392715332dbf9fe794043a822707b1dff70d98002efc431d7c8043336"}
Nov 03 12:39:50.250891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.250880    4546 scope.go:122] "RemoveContainer" containerID="a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081"
Nov 03 12:39:50.251105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.251088    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"
Nov 03 12:39:50.267399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.267377    4546 scope.go:122] "RemoveContainer" containerID="a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081"
Nov 03 12:39:50.267806 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:39:50.267786    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081\": container with ID starting with a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081 not found: ID does not exist" containerID="a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081"
Nov 03 12:39:50.267870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.267814    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081"} err="failed to get container status \"a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081\": rpc error: code = NotFound desc = could not find container \"a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081\": container with ID starting with a1df48226a850a1af547fdf72d124b168fcd3542752a1772449b5045ff5f6081 not found: ID does not exist"
Nov 03 12:39:50.405676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.405614    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"]
Nov 03 12:39:50.408524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.408500    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-khtq2"]
Nov 03 12:39:50.573954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:39:50.573920    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" path="/var/lib/kubelet/pods/cf06f6d5-420f-4f6a-9103-ccff5cd0b392/volumes"
Nov 03 12:40:07.652409 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:40:07.652370    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:40:37.652108 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:40:37.652075    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:40:52.779680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.779646    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"]
Nov 03 12:40:52.780283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.780263    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy"
Nov 03 12:40:52.780283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.780283    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy"
Nov 03 12:40:52.780425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.780414    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf06f6d5-420f-4f6a-9103-ccff5cd0b392" containerName="istio-proxy"
Nov 03 12:40:52.783951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.783928    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.834046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.834024    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:40:52.875136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.875106    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.875306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.875213    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7n8d\" (UniqueName: \"kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.875306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.875260    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.880261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.880241    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:40:52.880261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.880242    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:40:52.880261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.880242    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:40:52.883260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.883242    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"]
Nov 03 12:40:52.976137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.976106    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.976268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.976240    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.976367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.976349    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7n8d\" (UniqueName: \"kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.976982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.976946    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:52.978605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:52.978584    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:53.029354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:53.029329    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7n8d\" (UniqueName: \"kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d\") pod \"maas-api-revoke-sub-4d948386-lz45r\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:53.099179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:53.099101    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:53.463000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:53.462976    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"]
Nov 03 12:40:53.464128 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:40:53.464101    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59436701_d4cc_4fc5_9d0b_c9aeb383dffc.slice/crio-d380797a5cc34c190e5a68157b34910695cf4fd0867b09559d6608763e6c2490 WatchSource:0}: Error finding container d380797a5cc34c190e5a68157b34910695cf4fd0867b09559d6608763e6c2490: Status 404 returned error can't find the container with id d380797a5cc34c190e5a68157b34910695cf4fd0867b09559d6608763e6c2490
Nov 03 12:40:53.649675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:53.649634    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r" event={"ID":"59436701-d4cc-4fc5-9d0b-c9aeb383dffc","Type":"ContainerStarted","Data":"d380797a5cc34c190e5a68157b34910695cf4fd0867b09559d6608763e6c2490"}
Nov 03 12:40:54.657283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:54.657204    4546 generic.go:356] "Generic (PLEG): container finished" podID="59436701-d4cc-4fc5-9d0b-c9aeb383dffc" containerID="e0a4169953f5e42e4cc76b746194d2957a24b501d942068e001c364244c47232" exitCode=0
Nov 03 12:40:54.657644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:54.657282    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r" event={"ID":"59436701-d4cc-4fc5-9d0b-c9aeb383dffc","Type":"ContainerDied","Data":"e0a4169953f5e42e4cc76b746194d2957a24b501d942068e001c364244c47232"}
Nov 03 12:40:55.821722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.821697    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:55.940606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.940518    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca\") pod \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") "
Nov 03 12:40:55.942898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.942871    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca\") pod \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") "
Nov 03 12:40:55.943608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.943590    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d\" (UniqueName: \"kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d\") pod \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\" (UID: \"59436701-d4cc-4fc5-9d0b-c9aeb383dffc\") "
Nov 03 12:40:55.944606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.943518    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca" pod "59436701-d4cc-4fc5-9d0b-c9aeb383dffc" (UID: "59436701-d4cc-4fc5-9d0b-c9aeb383dffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:40:55.948668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.948612    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca" pod "59436701-d4cc-4fc5-9d0b-c9aeb383dffc" (UID: "59436701-d4cc-4fc5-9d0b-c9aeb383dffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:40:55.949211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:55.949188    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d" pod "59436701-d4cc-4fc5-9d0b-c9aeb383dffc" (UID: "59436701-d4cc-4fc5-9d0b-c9aeb383dffc"). InnerVolumeSpecName "kube-api-access-n7n8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:40:56.046019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:56.045984    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:56.046019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:56.046015    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:56.046019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:56.046026    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7n8d\" (UniqueName: \"kubernetes.io/projected/59436701-d4cc-4fc5-9d0b-c9aeb383dffc-kube-api-access-n7n8d\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:56.671680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:56.671641    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"
Nov 03 12:40:56.671909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:56.671641    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r" event={"ID":"59436701-d4cc-4fc5-9d0b-c9aeb383dffc","Type":"ContainerDied","Data":"d380797a5cc34c190e5a68157b34910695cf4fd0867b09559d6608763e6c2490"}
Nov 03 12:40:56.671909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:40:56.671788    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d380797a5cc34c190e5a68157b34910695cf4fd0867b09559d6608763e6c2490"
Nov 03 12:41:07.651374 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:41:07.651298    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:41:25.967367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:25.967335    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"]
Nov 03 12:41:25.968911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:25.967568    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy" containerID="cri-o://1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64" gracePeriod=30
Nov 03 12:41:31.245175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.245116    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:41:31.397074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397046    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397095    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397126    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397244    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397365    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397407    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397489    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397491    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:31.397563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397541    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:31.397718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397559    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397671    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:31.397718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397683    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy\") pod \"deb4b2db-141c-48d2-935d-11696737f7d6\" (UID: \"deb4b2db-141c-48d2-935d-11696737f7d6\") "
Nov 03 12:41:31.397718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.397691    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:31.398046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.398025    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:31.398811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.398672    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb4b2db-141c-48d2-935d-11696737f7d6-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.398811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.398700    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.398811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.398715    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.398811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.398728    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.398811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.398767    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.400458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.400426    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:41:31.402504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.402257    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:31.402504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.402369    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "kube-api-access-f5zr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:31.402504 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.402476    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy" pod "deb4b2db-141c-48d2-935d-11696737f7d6" (UID: "deb4b2db-141c-48d2-935d-11696737f7d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:31.500483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.500403    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zr5\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-kube-api-access-f5zr5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.500483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.500432    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb4b2db-141c-48d2-935d-11696737f7d6-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.500483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.500445    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb4b2db-141c-48d2-935d-11696737f7d6-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.500483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.500453    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb4b2db-141c-48d2-935d-11696737f7d6-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:31.908867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.908782    4546 generic.go:356] "Generic (PLEG): container finished" podID="deb4b2db-141c-48d2-935d-11696737f7d6" containerID="1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64" exitCode=0
Nov 03 12:41:31.908867 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.908845    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"
Nov 03 12:41:31.909075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.908846    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" event={"ID":"deb4b2db-141c-48d2-935d-11696737f7d6","Type":"ContainerDied","Data":"1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64"}
Nov 03 12:41:31.909075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.908967    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs" event={"ID":"deb4b2db-141c-48d2-935d-11696737f7d6","Type":"ContainerDied","Data":"3584245dece089b1a0a4e3bd1d570ca7e7ceafc451bc5321c6c73e504fbe7200"}
Nov 03 12:41:31.909075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.908993    4546 scope.go:122] "RemoveContainer" containerID="1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64"
Nov 03 12:41:31.919702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.919678    4546 scope.go:122] "RemoveContainer" containerID="69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"
Nov 03 12:41:31.928456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.928439    4546 scope.go:122] "RemoveContainer" containerID="1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64"
Nov 03 12:41:31.928720 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:41:31.928697    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64\": container with ID starting with 1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64 not found: ID does not exist" containerID="1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64"
Nov 03 12:41:31.928807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.928728    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64"} err="failed to get container status \"1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64\": rpc error: code = NotFound desc = could not find container \"1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64\": container with ID starting with 1c258e74f711b0974af9f9b8d2f350470684497d3e4538ac1850600fd6322d64 not found: ID does not exist"
Nov 03 12:41:31.928807 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.928762    4546 scope.go:122] "RemoveContainer" containerID="69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"
Nov 03 12:41:31.928994 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:41:31.928976    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1\": container with ID starting with 69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1 not found: ID does not exist" containerID="69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"
Nov 03 12:41:31.929037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:31.929000    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1"} err="failed to get container status \"69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1\": rpc error: code = NotFound desc = could not find container \"69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1\": container with ID starting with 69898325b371f679443b1bb3418faadb5323a727116fdefcd270faf35b99f3d1 not found: ID does not exist"
Nov 03 12:41:32.078613 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:32.078528    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"]
Nov 03 12:41:32.082301 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:32.082272    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4455f-openshift-default-5595d4ffcc-4lqqs"]
Nov 03 12:41:32.575345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:41:32.575305    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" path="/var/lib/kubelet/pods/deb4b2db-141c-48d2-935d-11696737f7d6/volumes"
Nov 03 12:41:37.652855 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:41:37.652817    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:42:07.651197 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:42:07.651164    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:42:22.133734 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.133679    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"]
Nov 03 12:42:22.134492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134469    4546 cpu_manager.go:414] "containerMap: removing container" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy"
Nov 03 12:42:22.134595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134494    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy"
Nov 03 12:42:22.134595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134518    4546 cpu_manager.go:414] "containerMap: removing container" podUID="59436701-d4cc-4fc5-9d0b-c9aeb383dffc" containerName="revoke-keys"
Nov 03 12:42:22.134595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134525    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59436701-d4cc-4fc5-9d0b-c9aeb383dffc" containerName="revoke-keys"
Nov 03 12:42:22.134787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134704    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy"
Nov 03 12:42:22.134787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134722    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="59436701-d4cc-4fc5-9d0b-c9aeb383dffc" containerName="revoke-keys"
Nov 03 12:42:22.134787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.134736    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy"
Nov 03 12:42:22.139687 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.139665    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.191002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.190965    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"]
Nov 03 12:42:22.233294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.233252    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:42:22.233481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.233340    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:42:22.233481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.233351    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:42:22.233481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.233414    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:42:22.261120 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.261091    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.261319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.261227    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.261319 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.261307    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667mf\" (UniqueName: \"kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.362845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.362807    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.363032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.362903    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667mf\" (UniqueName: \"kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.363032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.362984    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.363540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.363507    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.365448 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.365426    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.415630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.415556    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-667mf\" (UniqueName: \"kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf\") pod \"maas-api-revoke-sub-a2673e26-q59h8\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.453462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.453424    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:22.822577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:22.822548    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"]
Nov 03 12:42:22.823623 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:42:22.823596    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7c8ffd_8c04_41f3_9b15_b2b52b0253de.slice/crio-d19d80d60a3c9382123326d4cbac42ea4a03c33891ba7833c2b4dfceee6d77ef WatchSource:0}: Error finding container d19d80d60a3c9382123326d4cbac42ea4a03c33891ba7833c2b4dfceee6d77ef: Status 404 returned error can't find the container with id d19d80d60a3c9382123326d4cbac42ea4a03c33891ba7833c2b4dfceee6d77ef
Nov 03 12:42:23.252360 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:23.252322    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8" event={"ID":"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de","Type":"ContainerStarted","Data":"81670e61f9000d75cc31a0caea5c0c5512841d9bf6dd6b3699f4629df4403a06"}
Nov 03 12:42:23.252868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:23.252379    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8" event={"ID":"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de","Type":"ContainerStarted","Data":"d19d80d60a3c9382123326d4cbac42ea4a03c33891ba7833c2b4dfceee6d77ef"}
Nov 03 12:42:23.358170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:23.358120    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8" podStartSLOduration=1.358105079 podStartE2EDuration="1.358105079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:42:23.356737775 +0000 UTC m=+3084.928321701" watchObservedRunningTime="2026-11-03 12:42:23.358105079 +0000 UTC m=+3084.929688991"
Nov 03 12:42:24.260271 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:24.260244    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" containerID="81670e61f9000d75cc31a0caea5c0c5512841d9bf6dd6b3699f4629df4403a06" exitCode=0
Nov 03 12:42:24.260695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:24.260330    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8" event={"ID":"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de","Type":"ContainerDied","Data":"81670e61f9000d75cc31a0caea5c0c5512841d9bf6dd6b3699f4629df4403a06"}
Nov 03 12:42:25.422719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.422687    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:25.604775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.604669    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf\" (UniqueName: \"kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf\") pod \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") "
Nov 03 12:42:25.604950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.604837    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca\") pod \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") "
Nov 03 12:42:25.604950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.604887    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca\") pod \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\" (UID: \"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de\") "
Nov 03 12:42:25.605349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.605320    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca" pod "6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" (UID: "6d7c8ffd-8c04-41f3-9b15-b2b52b0253de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:42:25.607180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.607149    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca" pod "6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" (UID: "6d7c8ffd-8c04-41f3-9b15-b2b52b0253de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:42:25.607278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.607251    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf" pod "6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" (UID: "6d7c8ffd-8c04-41f3-9b15-b2b52b0253de"). InnerVolumeSpecName "kube-api-access-667mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:25.706227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.706187    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667mf\" (UniqueName: \"kubernetes.io/projected/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-kube-api-access-667mf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:25.706227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.706221    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:25.706227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:25.706230    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:26.274977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:26.274939    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8" event={"ID":"6d7c8ffd-8c04-41f3-9b15-b2b52b0253de","Type":"ContainerDied","Data":"d19d80d60a3c9382123326d4cbac42ea4a03c33891ba7833c2b4dfceee6d77ef"}
Nov 03 12:42:26.275144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:26.274996    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19d80d60a3c9382123326d4cbac42ea4a03c33891ba7833c2b4dfceee6d77ef"
Nov 03 12:42:26.275144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:26.274948    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"
Nov 03 12:42:32.832236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.832205    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"]
Nov 03 12:42:32.832821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.832806    4546 cpu_manager.go:414] "containerMap: removing container" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy"
Nov 03 12:42:32.832877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.832822    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb4b2db-141c-48d2-935d-11696737f7d6" containerName="istio-proxy"
Nov 03 12:42:32.832877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.832844    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" containerName="revoke-keys"
Nov 03 12:42:32.832877 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.832851    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" containerName="revoke-keys"
Nov 03 12:42:32.832983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.832952    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" containerName="revoke-keys"
Nov 03 12:42:32.836111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.836094    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.886465 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.886435    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:42:32.886614 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.886499    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:42:32.889182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.889158    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.889309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.889232    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.889418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.889398    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnm4\" (UniqueName: \"kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.932556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.932529    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:42:32.932719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.932565    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:42:32.935819 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.935797    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"]
Nov 03 12:42:32.990483 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.990449    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnm4\" (UniqueName: \"kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.990662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.990571    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.990662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.990611    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.991246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.991218    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:32.993134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:32.993107    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:33.043962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:33.043932    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnm4\" (UniqueName: \"kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4\") pod \"maas-api-revoke-sub-60ee0566-dstl8\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:33.150062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:33.149985    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:33.521837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:33.521808    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"]
Nov 03 12:42:33.522812 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:42:33.522783    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07677acf_8e3c_4756_9a03_243fdc1b9fca.slice/crio-c8f0b54bfa6157f5a231bf513e31fa274693be55d69ac09cbf028e19100c8c3e WatchSource:0}: Error finding container c8f0b54bfa6157f5a231bf513e31fa274693be55d69ac09cbf028e19100c8c3e: Status 404 returned error can't find the container with id c8f0b54bfa6157f5a231bf513e31fa274693be55d69ac09cbf028e19100c8c3e
Nov 03 12:42:34.323743 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:34.323707    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8" event={"ID":"07677acf-8e3c-4756-9a03-243fdc1b9fca","Type":"ContainerStarted","Data":"15ad5bc7722c87f37a84e1de26e8d5a1a062d554610ae1739ecd3e8ef1a0106b"}
Nov 03 12:42:34.324159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:34.323788    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8" event={"ID":"07677acf-8e3c-4756-9a03-243fdc1b9fca","Type":"ContainerStarted","Data":"c8f0b54bfa6157f5a231bf513e31fa274693be55d69ac09cbf028e19100c8c3e"}
Nov 03 12:42:34.429722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:34.429667    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8" podStartSLOduration=2.429649501 podStartE2EDuration="2.429649501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:42:34.428138188 +0000 UTC m=+3095.999722120" watchObservedRunningTime="2026-11-03 12:42:34.429649501 +0000 UTC m=+3096.001233414"
Nov 03 12:42:35.331174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:35.331142    4546 generic.go:356] "Generic (PLEG): container finished" podID="07677acf-8e3c-4756-9a03-243fdc1b9fca" containerID="15ad5bc7722c87f37a84e1de26e8d5a1a062d554610ae1739ecd3e8ef1a0106b" exitCode=0
Nov 03 12:42:35.331572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:35.331225    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8" event={"ID":"07677acf-8e3c-4756-9a03-243fdc1b9fca","Type":"ContainerDied","Data":"15ad5bc7722c87f37a84e1de26e8d5a1a062d554610ae1739ecd3e8ef1a0106b"}
Nov 03 12:42:36.495779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.495738    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:36.534878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.534846    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca\") pod \"07677acf-8e3c-4756-9a03-243fdc1b9fca\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") "
Nov 03 12:42:36.535073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.534941    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca\") pod \"07677acf-8e3c-4756-9a03-243fdc1b9fca\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") "
Nov 03 12:42:36.535128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.535114    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4\" (UniqueName: \"kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4\") pod \"07677acf-8e3c-4756-9a03-243fdc1b9fca\" (UID: \"07677acf-8e3c-4756-9a03-243fdc1b9fca\") "
Nov 03 12:42:36.535308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.535278    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca" pod "07677acf-8e3c-4756-9a03-243fdc1b9fca" (UID: "07677acf-8e3c-4756-9a03-243fdc1b9fca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:42:36.535668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.535648    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:36.537102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.537076    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca" pod "07677acf-8e3c-4756-9a03-243fdc1b9fca" (UID: "07677acf-8e3c-4756-9a03-243fdc1b9fca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:42:36.537317 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.537298    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4" pod "07677acf-8e3c-4756-9a03-243fdc1b9fca" (UID: "07677acf-8e3c-4756-9a03-243fdc1b9fca"). InnerVolumeSpecName "kube-api-access-6lnm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:36.637411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.637322    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07677acf-8e3c-4756-9a03-243fdc1b9fca-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:36.637411 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:36.637359    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lnm4\" (UniqueName: \"kubernetes.io/projected/07677acf-8e3c-4756-9a03-243fdc1b9fca-kube-api-access-6lnm4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:37.344343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:37.344309    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"
Nov 03 12:42:37.344343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:37.344343    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8" event={"ID":"07677acf-8e3c-4756-9a03-243fdc1b9fca","Type":"ContainerDied","Data":"c8f0b54bfa6157f5a231bf513e31fa274693be55d69ac09cbf028e19100c8c3e"}
Nov 03 12:42:37.344569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:37.344407    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f0b54bfa6157f5a231bf513e31fa274693be55d69ac09cbf028e19100c8c3e"
Nov 03 12:42:37.401979 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:37.401949    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"]
Nov 03 12:42:37.402158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:37.402139    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" containerID="cri-o://563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31" gracePeriod=30
Nov 03 12:42:37.651719 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:42:37.651596    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:42:42.537572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.537538    4546 patch_prober.go:29] interesting pod/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 12:42:42.537958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.537609    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 12:42:42.687184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.687159    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:42:42.828432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828397    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.828432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828434    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.828720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828548    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.828720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828652    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.828889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828736    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.828889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828779    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:42.828998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828917    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:42.828998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828952    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.828998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.828970    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:42:42.829144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829022    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.829144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829042    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.829144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829071    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs\") pod \"defce8d6-9532-451e-8ffc-7335e035cf83\" (UID: \"defce8d6-9532-451e-8ffc-7335e035cf83\") "
Nov 03 12:42:42.829323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829211    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:42.829402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829371    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:42.829875 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829849    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.830011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829878    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defce8d6-9532-451e-8ffc-7335e035cf83-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.830011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829896    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.830011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829910    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.830011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.829923    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.831365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.831261    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:42.831365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.831333    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:42.831365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.831342    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "kube-api-access-mgpw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:42.831531 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.831442    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo" pod "defce8d6-9532-451e-8ffc-7335e035cf83" (UID: "defce8d6-9532-451e-8ffc-7335e035cf83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:42:42.930966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.930881    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgpw5\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-kube-api-access-mgpw5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.930966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.930913    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defce8d6-9532-451e-8ffc-7335e035cf83-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.930966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.930923    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defce8d6-9532-451e-8ffc-7335e035cf83-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:42.930966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:42.930931    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defce8d6-9532-451e-8ffc-7335e035cf83-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:43.381583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.381550    4546 generic.go:356] "Generic (PLEG): container finished" podID="defce8d6-9532-451e-8ffc-7335e035cf83" containerID="563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31" exitCode=0
Nov 03 12:42:43.381742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.381607    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"
Nov 03 12:42:43.381742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.381628    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" event={"ID":"defce8d6-9532-451e-8ffc-7335e035cf83","Type":"ContainerDied","Data":"563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31"}
Nov 03 12:42:43.381742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.381687    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz" event={"ID":"defce8d6-9532-451e-8ffc-7335e035cf83","Type":"ContainerDied","Data":"cb4d7e27b9fa213c0e76f40538d8fa054fb7b2d87a59089241f5a76f7de07149"}
Nov 03 12:42:43.381742 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.381709    4546 scope.go:122] "RemoveContainer" containerID="563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31"
Nov 03 12:42:43.392934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.392909    4546 scope.go:122] "RemoveContainer" containerID="24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"
Nov 03 12:42:43.405428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.405408    4546 scope.go:122] "RemoveContainer" containerID="563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31"
Nov 03 12:42:43.405674 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:42:43.405655    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31\": container with ID starting with 563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31 not found: ID does not exist" containerID="563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31"
Nov 03 12:42:43.405728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.405683    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31"} err="failed to get container status \"563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31\": rpc error: code = NotFound desc = could not find container \"563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31\": container with ID starting with 563f92ef8c80ce7dff0757118a720304e9525c96db0a361e398b364f982f5f31 not found: ID does not exist"
Nov 03 12:42:43.405728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.405706    4546 scope.go:122] "RemoveContainer" containerID="24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"
Nov 03 12:42:43.405959 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:42:43.405943    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57\": container with ID starting with 24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57 not found: ID does not exist" containerID="24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"
Nov 03 12:42:43.405997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.405965    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57"} err="failed to get container status \"24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57\": rpc error: code = NotFound desc = could not find container \"24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57\": container with ID starting with 24723f77cde81afcb11b06a9e5934bcb0d00e654f38461ffb43985dbb4332f57 not found: ID does not exist"
Nov 03 12:42:43.580984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.580954    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"]
Nov 03 12:42:43.627673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:43.627640    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3176d9-openshift-default-6664c679bf-m2pgz"]
Nov 03 12:42:44.573296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:44.573260    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" path="/var/lib/kubelet/pods/defce8d6-9532-451e-8ffc-7335e035cf83/volumes"
Nov 03 12:42:45.942893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:45.942864    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"]
Nov 03 12:42:45.943287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:45.943047    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" containerID="cri-o://3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764" gracePeriod=30
Nov 03 12:42:46.709948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:46.709917    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"]
Nov 03 12:42:49.668597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:49.668563    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:42:49.668987 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:49.668627    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:42:51.219969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.219943    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:42:51.257183 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257154    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257203    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257222    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257326    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257371    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257425    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257462    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257499    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:51.257685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257539    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.257685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257573    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:51.257685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257619    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket\") pod \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\" (UID: \"a71da09f-6993-4c88-9e9f-ac5059b9a6fd\") "
Nov 03 12:42:51.258062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257726    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:51.258062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257780    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:42:51.258062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.257968    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:51.258250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.258182    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.258250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.258202    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.258250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.258215    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.258250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.258229    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.258733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.258260    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.259595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.259575    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:42:51.259727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.259701    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "kube-api-access-jk99x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:51.260055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.260032    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:51.260199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.260184    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy" pod "a71da09f-6993-4c88-9e9f-ac5059b9a6fd" (UID: "a71da09f-6993-4c88-9e9f-ac5059b9a6fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:42:51.359956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.359915    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.359956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.359943    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk99x\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-kube-api-access-jk99x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.359956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.359954    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.359956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.359962    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71da09f-6993-4c88-9e9f-ac5059b9a6fd-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:51.440017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.439982    4546 generic.go:356] "Generic (PLEG): container finished" podID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerID="3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764" exitCode=0
Nov 03 12:42:51.440208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.440030    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"
Nov 03 12:42:51.440208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.440061    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" event={"ID":"a71da09f-6993-4c88-9e9f-ac5059b9a6fd","Type":"ContainerDied","Data":"3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764"}
Nov 03 12:42:51.440208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.440111    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4" event={"ID":"a71da09f-6993-4c88-9e9f-ac5059b9a6fd","Type":"ContainerDied","Data":"cfa7d12370f08c0de9158319521ef09942385b21712bf4286d4aae84499d48d2"}
Nov 03 12:42:51.440208 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.440140    4546 scope.go:122] "RemoveContainer" containerID="3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764"
Nov 03 12:42:51.450672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.450654    4546 scope.go:122] "RemoveContainer" containerID="3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764"
Nov 03 12:42:51.450958 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:42:51.450940    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764\": container with ID starting with 3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764 not found: ID does not exist" containerID="3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764"
Nov 03 12:42:51.451067 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.450967    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764"} err="failed to get container status \"3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764\": rpc error: code = NotFound desc = could not find container \"3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764\": container with ID starting with 3cf63ad403b11e344e1d411301ffa871dbfe91b4b6db903c2d88fa91946f7764 not found: ID does not exist"
Nov 03 12:42:51.593456 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.593379    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"]
Nov 03 12:42:51.597679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:51.597648    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-4thj4"]
Nov 03 12:42:52.577836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:52.577799    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" path="/var/lib/kubelet/pods/a71da09f-6993-4c88-9e9f-ac5059b9a6fd/volumes"
Nov 03 12:42:58.970325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.970283    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"]
Nov 03 12:42:58.971169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971145    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971170    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971190    4546 cpu_manager.go:414] "containerMap: removing container" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971197    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971226    4546 cpu_manager.go:414] "containerMap: removing container" podUID="07677acf-8e3c-4756-9a03-243fdc1b9fca" containerName="revoke-keys"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971234    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07677acf-8e3c-4756-9a03-243fdc1b9fca" containerName="revoke-keys"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971252    4546 cpu_manager.go:414] "containerMap: removing container" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy"
Nov 03 12:42:58.971288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971258    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy"
Nov 03 12:42:58.971656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971415    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71da09f-6993-4c88-9e9f-ac5059b9a6fd" containerName="istio-proxy"
Nov 03 12:42:58.971656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971429    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy"
Nov 03 12:42:58.971656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971443    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="07677acf-8e3c-4756-9a03-243fdc1b9fca" containerName="revoke-keys"
Nov 03 12:42:58.971656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.971457    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="defce8d6-9532-451e-8ffc-7335e035cf83" containerName="istio-proxy"
Nov 03 12:42:58.974939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:58.974919    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.024158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.024127    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:42:59.051246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.051217    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.051417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.051344    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.051525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.051512    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j67b\" (UniqueName: \"kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.070385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.070357    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:42:59.070536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.070436    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:42:59.070536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.070464    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:42:59.075469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.075447    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"]
Nov 03 12:42:59.152485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.152419    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j67b\" (UniqueName: \"kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.152485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.152473    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.152823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.152589    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.153530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.153499    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.155332 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.155308    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.205345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.205319    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j67b\" (UniqueName: \"kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b\") pod \"maas-api-revoke-sub-e431771c-ngtc5\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.287945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.287862    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:42:59.656898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:42:59.656868    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"]
Nov 03 12:42:59.658197 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:42:59.658173    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7191e0c_201a_4188_8509_7bffe2d42858.slice/crio-cd61232e3318009dec2ff78fb43fc099834322de3eb258d3df830949bb9c34bb WatchSource:0}: Error finding container cd61232e3318009dec2ff78fb43fc099834322de3eb258d3df830949bb9c34bb: Status 404 returned error can't find the container with id cd61232e3318009dec2ff78fb43fc099834322de3eb258d3df830949bb9c34bb
Nov 03 12:43:00.498727 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:00.498682    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5" event={"ID":"f7191e0c-201a-4188-8509-7bffe2d42858","Type":"ContainerStarted","Data":"b2231ce2899a6bb853dfd90c66b4022203f3ff499e18825534bd1146c8c8cacb"}
Nov 03 12:43:00.499246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:00.498737    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5" event={"ID":"f7191e0c-201a-4188-8509-7bffe2d42858","Type":"ContainerStarted","Data":"cd61232e3318009dec2ff78fb43fc099834322de3eb258d3df830949bb9c34bb"}
Nov 03 12:43:00.604766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:00.604700    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5" podStartSLOduration=2.60468493 podStartE2EDuration="2.60468493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:00.602622188 +0000 UTC m=+3122.174206113" watchObservedRunningTime="2026-11-03 12:43:00.60468493 +0000 UTC m=+3122.176268841"
Nov 03 12:43:01.506778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:01.506666    4546 generic.go:356] "Generic (PLEG): container finished" podID="f7191e0c-201a-4188-8509-7bffe2d42858" containerID="b2231ce2899a6bb853dfd90c66b4022203f3ff499e18825534bd1146c8c8cacb" exitCode=0
Nov 03 12:43:01.507156 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:01.506776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5" event={"ID":"f7191e0c-201a-4188-8509-7bffe2d42858","Type":"ContainerDied","Data":"b2231ce2899a6bb853dfd90c66b4022203f3ff499e18825534bd1146c8c8cacb"}
Nov 03 12:43:01.735833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:01.735790    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc" podUID="6141c84b-9c1f-41be-a999-835d394c5c5b" containerName="main" containerID="cri-o://bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2" gracePeriod=45
Nov 03 12:43:02.025701 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.025675    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:43:02.090801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.090762    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir\") pod \"6141c84b-9c1f-41be-a999-835d394c5c5b\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") "
Nov 03 12:43:02.090988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.090948    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv\" (UniqueName: \"kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv\") pod \"6141c84b-9c1f-41be-a999-835d394c5c5b\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") "
Nov 03 12:43:02.090988 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.090980    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache\") pod \"6141c84b-9c1f-41be-a999-835d394c5c5b\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") "
Nov 03 12:43:02.091102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091003    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home\") pod \"6141c84b-9c1f-41be-a999-835d394c5c5b\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") "
Nov 03 12:43:02.091102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091013    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir" pod "6141c84b-9c1f-41be-a999-835d394c5c5b" (UID: "6141c84b-9c1f-41be-a999-835d394c5c5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:02.091102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091079    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm\") pod \"6141c84b-9c1f-41be-a999-835d394c5c5b\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") "
Nov 03 12:43:02.091102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091099    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs\") pod \"6141c84b-9c1f-41be-a999-835d394c5c5b\" (UID: \"6141c84b-9c1f-41be-a999-835d394c5c5b\") "
Nov 03 12:43:02.091326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091236    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache" pod "6141c84b-9c1f-41be-a999-835d394c5c5b" (UID: "6141c84b-9c1f-41be-a999-835d394c5c5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:02.091326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091312    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home" pod "6141c84b-9c1f-41be-a999-835d394c5c5b" (UID: "6141c84b-9c1f-41be-a999-835d394c5c5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:02.091664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091643    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.091664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091665    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.092026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.091674    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.093455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.093432    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm" pod "6141c84b-9c1f-41be-a999-835d394c5c5b" (UID: "6141c84b-9c1f-41be-a999-835d394c5c5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:02.093540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.093510    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs" pod "6141c84b-9c1f-41be-a999-835d394c5c5b" (UID: "6141c84b-9c1f-41be-a999-835d394c5c5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:43:02.093581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.093538    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv" pod "6141c84b-9c1f-41be-a999-835d394c5c5b" (UID: "6141c84b-9c1f-41be-a999-835d394c5c5b"). InnerVolumeSpecName "kube-api-access-8dprv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:02.192773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.192663    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dprv\" (UniqueName: \"kubernetes.io/projected/6141c84b-9c1f-41be-a999-835d394c5c5b-kube-api-access-8dprv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.192773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.192696    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6141c84b-9c1f-41be-a999-835d394c5c5b-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.192773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.192711    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6141c84b-9c1f-41be-a999-835d394c5c5b-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.515244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.515211    4546 generic.go:356] "Generic (PLEG): container finished" podID="6141c84b-9c1f-41be-a999-835d394c5c5b" containerID="bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2" exitCode=0
Nov 03 12:43:02.515683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.515257    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"
Nov 03 12:43:02.515683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.515279    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc" event={"ID":"6141c84b-9c1f-41be-a999-835d394c5c5b","Type":"ContainerDied","Data":"bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2"}
Nov 03 12:43:02.515683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.515342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc" event={"ID":"6141c84b-9c1f-41be-a999-835d394c5c5b","Type":"ContainerDied","Data":"47e92ebe755d2c6dba37213ebbb278e1e95dd19e1eecdd66079d228d2141814b"}
Nov 03 12:43:02.515683 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.515361    4546 scope.go:122] "RemoveContainer" containerID="bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2"
Nov 03 12:43:02.527557 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.527531    4546 scope.go:122] "RemoveContainer" containerID="bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2"
Nov 03 12:43:02.527896 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:43:02.527873    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2\": container with ID starting with bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2 not found: ID does not exist" containerID="bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2"
Nov 03 12:43:02.527975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.527907    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2"} err="failed to get container status \"bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2\": rpc error: code = NotFound desc = could not find container \"bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2\": container with ID starting with bf1cad0fa8f1e34163b40fd9640c09f51be29051f27cb6a4055bbfdb776708c2 not found: ID does not exist"
Nov 03 12:43:02.657703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.657680    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:43:02.669279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.669258    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"]
Nov 03 12:43:02.674415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.674393    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ab27b8/auth-test-model-ab27b8-kserve-7459dc6469-5nzhc"]
Nov 03 12:43:02.802109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.802020    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca\") pod \"f7191e0c-201a-4188-8509-7bffe2d42858\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") "
Nov 03 12:43:02.802244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.802110    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b\" (UniqueName: \"kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b\") pod \"f7191e0c-201a-4188-8509-7bffe2d42858\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") "
Nov 03 12:43:02.802244 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.802201    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca\") pod \"f7191e0c-201a-4188-8509-7bffe2d42858\" (UID: \"f7191e0c-201a-4188-8509-7bffe2d42858\") "
Nov 03 12:43:02.802767 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.802703    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca" pod "f7191e0c-201a-4188-8509-7bffe2d42858" (UID: "f7191e0c-201a-4188-8509-7bffe2d42858"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:43:02.804735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.804703    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca" pod "f7191e0c-201a-4188-8509-7bffe2d42858" (UID: "f7191e0c-201a-4188-8509-7bffe2d42858"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:43:02.804735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.804718    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b" pod "f7191e0c-201a-4188-8509-7bffe2d42858" (UID: "f7191e0c-201a-4188-8509-7bffe2d42858"). InnerVolumeSpecName "kube-api-access-6j67b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:02.904354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.904318    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j67b\" (UniqueName: \"kubernetes.io/projected/f7191e0c-201a-4188-8509-7bffe2d42858-kube-api-access-6j67b\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.904354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.904350    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:02.904354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:02.904361    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7191e0c-201a-4188-8509-7bffe2d42858-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:03.523960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:03.523930    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"
Nov 03 12:43:03.524364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:03.523958    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5" event={"ID":"f7191e0c-201a-4188-8509-7bffe2d42858","Type":"ContainerDied","Data":"cd61232e3318009dec2ff78fb43fc099834322de3eb258d3df830949bb9c34bb"}
Nov 03 12:43:03.524364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:03.524033    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd61232e3318009dec2ff78fb43fc099834322de3eb258d3df830949bb9c34bb"
Nov 03 12:43:04.574609 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:04.574574    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6141c84b-9c1f-41be-a999-835d394c5c5b" path="/var/lib/kubelet/pods/6141c84b-9c1f-41be-a999-835d394c5c5b/volumes"
Nov 03 12:43:06.162900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:06.162871    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"]
Nov 03 12:43:07.650847 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:43:07.650803    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:43:09.695759 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.695722    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"]
Nov 03 12:43:09.696311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.696296    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f7191e0c-201a-4188-8509-7bffe2d42858" containerName="revoke-keys"
Nov 03 12:43:09.696359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.696313    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7191e0c-201a-4188-8509-7bffe2d42858" containerName="revoke-keys"
Nov 03 12:43:09.696359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.696340    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6141c84b-9c1f-41be-a999-835d394c5c5b" containerName="main"
Nov 03 12:43:09.696359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.696346    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6141c84b-9c1f-41be-a999-835d394c5c5b" containerName="main"
Nov 03 12:43:09.696460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.696430    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6141c84b-9c1f-41be-a999-835d394c5c5b" containerName="main"
Nov 03 12:43:09.696460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.696444    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7191e0c-201a-4188-8509-7bffe2d42858" containerName="revoke-keys"
Nov 03 12:43:09.700780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.700738    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.749439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.749414    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:43:09.749578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.749454    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:43:09.796030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.796002    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:43:09.796030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.796005    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:43:09.801126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.801101    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"]
Nov 03 12:43:09.894667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.894632    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.894859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.894696    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plxb\" (UniqueName: \"kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.894859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.894821    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.996558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.996525    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.996735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.996569    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plxb\" (UniqueName: \"kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.996735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.996614    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.997275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.997240    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:09.999112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:09.999093    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:10.049450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:10.049425    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plxb\" (UniqueName: \"kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb\") pod \"maas-api-revoke-sub-dae09438-fvw6l\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:10.314420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:10.314321    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:10.678883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:10.678860    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"]
Nov 03 12:43:10.680017 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:43:10.679987    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fcfb08_31ca_4f36_a229_be7d9cc5aa8a.slice/crio-3f3ffd07fc62ed24415f38a3909545dc1f0deb001e7ed9b2f7e4bb34be6ea06a WatchSource:0}: Error finding container 3f3ffd07fc62ed24415f38a3909545dc1f0deb001e7ed9b2f7e4bb34be6ea06a: Status 404 returned error can't find the container with id 3f3ffd07fc62ed24415f38a3909545dc1f0deb001e7ed9b2f7e4bb34be6ea06a
Nov 03 12:43:11.575863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:11.575832    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l" event={"ID":"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a","Type":"ContainerStarted","Data":"9c1613b74c527d425351dc6bb61ad63ffea90af175bf83f06c517a3d4dbfe1cb"}
Nov 03 12:43:11.576364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:11.575891    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l" event={"ID":"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a","Type":"ContainerStarted","Data":"3f3ffd07fc62ed24415f38a3909545dc1f0deb001e7ed9b2f7e4bb34be6ea06a"}
Nov 03 12:43:11.684894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:11.684839    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l" podStartSLOduration=2.6848248740000002 podStartE2EDuration="2.684824874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:11.68216481 +0000 UTC m=+3133.253748743" watchObservedRunningTime="2026-11-03 12:43:11.684824874 +0000 UTC m=+3133.256408785"
Nov 03 12:43:12.581956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:12.581923    4546 generic.go:356] "Generic (PLEG): container finished" podID="46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" containerID="9c1613b74c527d425351dc6bb61ad63ffea90af175bf83f06c517a3d4dbfe1cb" exitCode=0
Nov 03 12:43:12.582379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:12.582006    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l" event={"ID":"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a","Type":"ContainerDied","Data":"9c1613b74c527d425351dc6bb61ad63ffea90af175bf83f06c517a3d4dbfe1cb"}
Nov 03 12:43:13.737775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.737728    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:13.842338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.842254    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb\" (UniqueName: \"kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb\") pod \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") "
Nov 03 12:43:13.842338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.842335    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca\") pod \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") "
Nov 03 12:43:13.842569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.842426    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca\") pod \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\" (UID: \"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a\") "
Nov 03 12:43:13.842878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.842852    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca" pod "46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" (UID: "46fcfb08-31ca-4f36-a229-be7d9cc5aa8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:43:13.844715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.844690    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb" pod "46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" (UID: "46fcfb08-31ca-4f36-a229-be7d9cc5aa8a"). InnerVolumeSpecName "kube-api-access-5plxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:13.844836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.844786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca" pod "46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" (UID: "46fcfb08-31ca-4f36-a229-be7d9cc5aa8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:43:13.943947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.943916    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:13.943947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.943943    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:13.943947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:13.943953    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plxb\" (UniqueName: \"kubernetes.io/projected/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a-kube-api-access-5plxb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:14.594526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:14.594492    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l" event={"ID":"46fcfb08-31ca-4f36-a229-be7d9cc5aa8a","Type":"ContainerDied","Data":"3f3ffd07fc62ed24415f38a3909545dc1f0deb001e7ed9b2f7e4bb34be6ea06a"}
Nov 03 12:43:14.594526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:14.594521    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"
Nov 03 12:43:14.594774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:14.594537    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3ffd07fc62ed24415f38a3909545dc1f0deb001e7ed9b2f7e4bb34be6ea06a"
Nov 03 12:43:21.184074 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.184035    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j" podUID="f382114d-4d9e-44bd-aff6-e6d651c11803" containerName="main" containerID="cri-o://ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52" gracePeriod=45
Nov 03 12:43:21.468079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.468053    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:43:21.531540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531491    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache\") pod \"f382114d-4d9e-44bd-aff6-e6d651c11803\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") "
Nov 03 12:43:21.531730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531572    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm\") pod \"f382114d-4d9e-44bd-aff6-e6d651c11803\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") "
Nov 03 12:43:21.531730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531623    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs\" (UniqueName: \"kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs\") pod \"f382114d-4d9e-44bd-aff6-e6d651c11803\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") "
Nov 03 12:43:21.531730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531654    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home\") pod \"f382114d-4d9e-44bd-aff6-e6d651c11803\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") "
Nov 03 12:43:21.531730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531695    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc\") pod \"f382114d-4d9e-44bd-aff6-e6d651c11803\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") "
Nov 03 12:43:21.532031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531784    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache" pod "f382114d-4d9e-44bd-aff6-e6d651c11803" (UID: "f382114d-4d9e-44bd-aff6-e6d651c11803"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:21.532031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.531852    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir\") pod \"f382114d-4d9e-44bd-aff6-e6d651c11803\" (UID: \"f382114d-4d9e-44bd-aff6-e6d651c11803\") "
Nov 03 12:43:21.532031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.532008    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home" pod "f382114d-4d9e-44bd-aff6-e6d651c11803" (UID: "f382114d-4d9e-44bd-aff6-e6d651c11803"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:21.532293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.532271    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir" pod "f382114d-4d9e-44bd-aff6-e6d651c11803" (UID: "f382114d-4d9e-44bd-aff6-e6d651c11803"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:21.532439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.532423    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.532506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.532443    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.532506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.532453    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.534792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.534739    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm" pod "f382114d-4d9e-44bd-aff6-e6d651c11803" (UID: "f382114d-4d9e-44bd-aff6-e6d651c11803"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:21.534792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.534784    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc" pod "f382114d-4d9e-44bd-aff6-e6d651c11803" (UID: "f382114d-4d9e-44bd-aff6-e6d651c11803"). InnerVolumeSpecName "kube-api-access-zz2mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:21.534947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.534863    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs" pod "f382114d-4d9e-44bd-aff6-e6d651c11803" (UID: "f382114d-4d9e-44bd-aff6-e6d651c11803"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:43:21.633212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.633184    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f382114d-4d9e-44bd-aff6-e6d651c11803-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.633212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.633209    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f382114d-4d9e-44bd-aff6-e6d651c11803-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.633423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.633220    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/f382114d-4d9e-44bd-aff6-e6d651c11803-kube-api-access-zz2mc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.641724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.641694    4546 generic.go:356] "Generic (PLEG): container finished" podID="f382114d-4d9e-44bd-aff6-e6d651c11803" containerID="ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52" exitCode=0
Nov 03 12:43:21.641903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.641827    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j" event={"ID":"f382114d-4d9e-44bd-aff6-e6d651c11803","Type":"ContainerDied","Data":"ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52"}
Nov 03 12:43:21.641903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.641833    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"
Nov 03 12:43:21.641903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.641868    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j" event={"ID":"f382114d-4d9e-44bd-aff6-e6d651c11803","Type":"ContainerDied","Data":"f063f3d12cdb3ccb7397db4331efdb21d1db60b2fa5e8721f8e6aa89fde81152"}
Nov 03 12:43:21.641903 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.641890    4546 scope.go:122] "RemoveContainer" containerID="ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52"
Nov 03 12:43:21.652644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.652625    4546 scope.go:122] "RemoveContainer" containerID="ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52"
Nov 03 12:43:21.652971 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:43:21.652950    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52\": container with ID starting with ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52 not found: ID does not exist" containerID="ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52"
Nov 03 12:43:21.653059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.652979    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52"} err="failed to get container status \"ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52\": rpc error: code = NotFound desc = could not find container \"ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52\": container with ID starting with ef1d9ad6bbac58a2a16a726412b8669b79960b7ae5856d7abfa8b32a8f8d8a52 not found: ID does not exist"
Nov 03 12:43:21.800552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.800520    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"]
Nov 03 12:43:21.803678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:21.803655    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-743d9b/auth-test-model-743d9b-kserve-5599f6b469-clb5j"]
Nov 03 12:43:22.573236 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:22.573205    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f382114d-4d9e-44bd-aff6-e6d651c11803" path="/var/lib/kubelet/pods/f382114d-4d9e-44bd-aff6-e6d651c11803/volumes"
Nov 03 12:43:24.069032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069001    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"]
Nov 03 12:43:24.069607 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069590    4546 cpu_manager.go:414] "containerMap: removing container" podUID="46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" containerName="revoke-keys"
Nov 03 12:43:24.069658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069611    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" containerName="revoke-keys"
Nov 03 12:43:24.069698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069659    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f382114d-4d9e-44bd-aff6-e6d651c11803" containerName="main"
Nov 03 12:43:24.069698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069664    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f382114d-4d9e-44bd-aff6-e6d651c11803" containerName="main"
Nov 03 12:43:24.069811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069771    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f382114d-4d9e-44bd-aff6-e6d651c11803" containerName="main"
Nov 03 12:43:24.069811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.069782    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" containerName="revoke-keys"
Nov 03 12:43:24.074676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.074658    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.123255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.123213    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-02f899-openshift-default-dockercfg-gd8hl\""
Nov 03 12:43:24.163363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163324    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.163596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163450    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.163596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163500    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.163596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163532    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.163792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163598    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.163792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163734    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.163972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.163947    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.164166 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.164148    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmtgl\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.164262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.164189    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.173119 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.173093    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"]
Nov 03 12:43:24.264889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.264857    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmtgl\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.264910    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265029    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265083 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265075    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265109    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265141    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265169    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.265282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265203    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.266137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265286    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.266137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265403    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.266137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265494    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.266137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265714    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.266137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.265987    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.266137 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.266008    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.267762 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.267722    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.268018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.267998    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.317644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.317609    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.317792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.317656    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmtgl\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl\") pod \"e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") " pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.387794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.387715    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:24.805345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.805317    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"]
Nov 03 12:43:24.805899 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:43:24.805876    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323fc1aa_b75a_4423_a181_5ffc6b801396.slice/crio-3c188ab613e1377c40d929d001afc1604f574b5857ae513ac522c42cf947e747 WatchSource:0}: Error finding container 3c188ab613e1377c40d929d001afc1604f574b5857ae513ac522c42cf947e747: Status 404 returned error can't find the container with id 3c188ab613e1377c40d929d001afc1604f574b5857ae513ac522c42cf947e747
Nov 03 12:43:24.808539 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.808505    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:43:24.808660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.808571    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:43:24.808660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:24.808600    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:43:25.705938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:25.705901    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" event={"ID":"323fc1aa-b75a-4423-a181-5ffc6b801396","Type":"ContainerStarted","Data":"ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"}
Nov 03 12:43:25.706471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:25.705994    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" event={"ID":"323fc1aa-b75a-4423-a181-5ffc6b801396","Type":"ContainerStarted","Data":"3c188ab613e1377c40d929d001afc1604f574b5857ae513ac522c42cf947e747"}
Nov 03 12:43:25.815972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:25.815926    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podStartSLOduration=1.8159112830000002 podStartE2EDuration="1.815911283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:25.813612505 +0000 UTC m=+3147.385196465" watchObservedRunningTime="2026-11-03 12:43:25.815911283 +0000 UTC m=+3147.387495194"
Nov 03 12:43:26.388121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:26.388088    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:26.394076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:26.394049    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:26.712532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:26.712455    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:26.713342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:26.713324    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:43:32.404464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.404422    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"]
Nov 03 12:43:32.408638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.408615    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.501631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.501596    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc48\" (UniqueName: \"kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.501845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.501650    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.501900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.501845    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.504946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.504914    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:43:32.504946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.504936    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:43:32.504946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.504931    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:43:32.504946 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.504954    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:43:32.508402 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.508383    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"]
Nov 03 12:43:32.603226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.603187    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.603425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.603388    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc48\" (UniqueName: \"kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.603513 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.603490    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.604180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.604159    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.605849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.605825    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.656673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.656602    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc48\" (UniqueName: \"kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48\") pod \"maas-api-revoke-sub-b18a55b2-9rgl4\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:32.722741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:32.722701    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:33.089372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:33.089342    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"]
Nov 03 12:43:33.091301 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:43:33.091272    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4c187a_7c5a_414e_8694_2d59a5005774.slice/crio-6345355e128fdeef43d1b6fcc5ba1293da46bd38601b9dd5717db3de9190b01e WatchSource:0}: Error finding container 6345355e128fdeef43d1b6fcc5ba1293da46bd38601b9dd5717db3de9190b01e: Status 404 returned error can't find the container with id 6345355e128fdeef43d1b6fcc5ba1293da46bd38601b9dd5717db3de9190b01e
Nov 03 12:43:33.758847 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:33.758809    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4" event={"ID":"5c4c187a-7c5a-414e-8694-2d59a5005774","Type":"ContainerStarted","Data":"4b128d16761124668963dc85c9214f2b92cdaab223a01378d3065383f3f2a8a9"}
Nov 03 12:43:33.759283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:33.758877    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4" event={"ID":"5c4c187a-7c5a-414e-8694-2d59a5005774","Type":"ContainerStarted","Data":"6345355e128fdeef43d1b6fcc5ba1293da46bd38601b9dd5717db3de9190b01e"}
Nov 03 12:43:33.865474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:33.865410    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4" podStartSLOduration=1.865391309 podStartE2EDuration="1.865391309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:33.863424515 +0000 UTC m=+3155.435008446" watchObservedRunningTime="2026-11-03 12:43:33.865391309 +0000 UTC m=+3155.436975222"
Nov 03 12:43:34.766065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:34.766030    4546 generic.go:356] "Generic (PLEG): container finished" podID="5c4c187a-7c5a-414e-8694-2d59a5005774" containerID="4b128d16761124668963dc85c9214f2b92cdaab223a01378d3065383f3f2a8a9" exitCode=0
Nov 03 12:43:34.766470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:34.766121    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4" event={"ID":"5c4c187a-7c5a-414e-8694-2d59a5005774","Type":"ContainerDied","Data":"4b128d16761124668963dc85c9214f2b92cdaab223a01378d3065383f3f2a8a9"}
Nov 03 12:43:35.920900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.920874    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:35.945041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.945003    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48\" (UniqueName: \"kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48\") pod \"5c4c187a-7c5a-414e-8694-2d59a5005774\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") "
Nov 03 12:43:35.945223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.945137    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca\") pod \"5c4c187a-7c5a-414e-8694-2d59a5005774\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") "
Nov 03 12:43:35.945223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.945166    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca\") pod \"5c4c187a-7c5a-414e-8694-2d59a5005774\" (UID: \"5c4c187a-7c5a-414e-8694-2d59a5005774\") "
Nov 03 12:43:35.945594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.945569    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca" pod "5c4c187a-7c5a-414e-8694-2d59a5005774" (UID: "5c4c187a-7c5a-414e-8694-2d59a5005774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:43:35.947303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.947276    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48" pod "5c4c187a-7c5a-414e-8694-2d59a5005774" (UID: "5c4c187a-7c5a-414e-8694-2d59a5005774"). InnerVolumeSpecName "kube-api-access-vrc48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:35.947394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:35.947358    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca" pod "5c4c187a-7c5a-414e-8694-2d59a5005774" (UID: "5c4c187a-7c5a-414e-8694-2d59a5005774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:43:36.047161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:36.047087    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc48\" (UniqueName: \"kubernetes.io/projected/5c4c187a-7c5a-414e-8694-2d59a5005774-kube-api-access-vrc48\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:36.047161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:36.047117    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:36.047161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:36.047127    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4c187a-7c5a-414e-8694-2d59a5005774-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:36.778944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:36.778915    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"
Nov 03 12:43:36.779130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:36.778951    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4" event={"ID":"5c4c187a-7c5a-414e-8694-2d59a5005774","Type":"ContainerDied","Data":"6345355e128fdeef43d1b6fcc5ba1293da46bd38601b9dd5717db3de9190b01e"}
Nov 03 12:43:36.779130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:36.779001    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6345355e128fdeef43d1b6fcc5ba1293da46bd38601b9dd5717db3de9190b01e"
Nov 03 12:43:37.651793 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:43:37.651741    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:43:43.100768 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.100725    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"]
Nov 03 12:43:43.101315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.101301    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5c4c187a-7c5a-414e-8694-2d59a5005774" containerName="revoke-keys"
Nov 03 12:43:43.101382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.101318    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4c187a-7c5a-414e-8694-2d59a5005774" containerName="revoke-keys"
Nov 03 12:43:43.101453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.101441    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4c187a-7c5a-414e-8694-2d59a5005774" containerName="revoke-keys"
Nov 03 12:43:43.107634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.107604    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.158891 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.158866    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"]
Nov 03 12:43:43.200842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.200816    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:43:43.201032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.200856    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:43:43.201032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.200883    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:43:43.202598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.202576    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:43:43.243275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.243246    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9prr\" (UniqueName: \"kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.243421 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.243374    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.243544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.243526    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.344961 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.344919    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.345116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.345013    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9prr\" (UniqueName: \"kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.345116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.345078    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.345792 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.345681    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.347400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.347377    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.398407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.398352    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9prr\" (UniqueName: \"kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr\") pod \"maas-api-revoke-sub-b62c3613-k6rbq\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.423598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.423579    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:43.790821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.790789    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"]
Nov 03 12:43:43.792310 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:43:43.792281    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9e2d3f_7447_45a1_a9d7_2495e535cd13.slice/crio-6d8a418ad4d9c9f86c85f9c65e01174bc6766db7b6a5eba2e6c7a8c2877a9959 WatchSource:0}: Error finding container 6d8a418ad4d9c9f86c85f9c65e01174bc6766db7b6a5eba2e6c7a8c2877a9959: Status 404 returned error can't find the container with id 6d8a418ad4d9c9f86c85f9c65e01174bc6766db7b6a5eba2e6c7a8c2877a9959
Nov 03 12:43:43.825072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:43.825041    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq" event={"ID":"0f9e2d3f-7447-45a1-a9d7-2495e535cd13","Type":"ContainerStarted","Data":"6d8a418ad4d9c9f86c85f9c65e01174bc6766db7b6a5eba2e6c7a8c2877a9959"}
Nov 03 12:43:44.833628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:44.833592    4546 generic.go:356] "Generic (PLEG): container finished" podID="0f9e2d3f-7447-45a1-a9d7-2495e535cd13" containerID="7dc72d06f72b1960832750d5615dd2cb9c2de3bbbe8bb7c39de53db0d251fbd7" exitCode=0
Nov 03 12:43:44.834045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:44.833679    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq" event={"ID":"0f9e2d3f-7447-45a1-a9d7-2495e535cd13","Type":"ContainerDied","Data":"7dc72d06f72b1960832750d5615dd2cb9c2de3bbbe8bb7c39de53db0d251fbd7"}
Nov 03 12:43:45.989974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:45.989950    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:46.081426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.081400    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca\") pod \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") "
Nov 03 12:43:46.081586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.081489    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr\" (UniqueName: \"kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr\") pod \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") "
Nov 03 12:43:46.081631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.081587    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca\") pod \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\" (UID: \"0f9e2d3f-7447-45a1-a9d7-2495e535cd13\") "
Nov 03 12:43:46.081827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.081800    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca" pod "0f9e2d3f-7447-45a1-a9d7-2495e535cd13" (UID: "0f9e2d3f-7447-45a1-a9d7-2495e535cd13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:43:46.082042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.082022    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:46.083786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.083763    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca" pod "0f9e2d3f-7447-45a1-a9d7-2495e535cd13" (UID: "0f9e2d3f-7447-45a1-a9d7-2495e535cd13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:43:46.083889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.083794    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr" pod "0f9e2d3f-7447-45a1-a9d7-2495e535cd13" (UID: "0f9e2d3f-7447-45a1-a9d7-2495e535cd13"). InnerVolumeSpecName "kube-api-access-w9prr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:46.183699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.183667    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:46.183699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.183693    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9prr\" (UniqueName: \"kubernetes.io/projected/0f9e2d3f-7447-45a1-a9d7-2495e535cd13-kube-api-access-w9prr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:46.846657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.846582    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"
Nov 03 12:43:46.846657 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.846615    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq" event={"ID":"0f9e2d3f-7447-45a1-a9d7-2495e535cd13","Type":"ContainerDied","Data":"6d8a418ad4d9c9f86c85f9c65e01174bc6766db7b6a5eba2e6c7a8c2877a9959"}
Nov 03 12:43:46.846872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:43:46.846661    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8a418ad4d9c9f86c85f9c65e01174bc6766db7b6a5eba2e6c7a8c2877a9959"
Nov 03 12:44:07.652544 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:44:07.652456    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:44:10.791933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:10.791903    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"]
Nov 03 12:44:25.813044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:25.812998    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh" podUID="45d4b012-69a9-4698-8c37-b70d5c0e6296" containerName="main" containerID="cri-o://e605645282ee08507ce5d076e7d788a8e6ff76a6127ce0842b93ed63841fd2ab" gracePeriod=45
Nov 03 12:44:25.992947 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:25.992679    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"]
Nov 03 12:44:26.108163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.108129    4546 generic.go:356] "Generic (PLEG): container finished" podID="45d4b012-69a9-4698-8c37-b70d5c0e6296" containerID="e605645282ee08507ce5d076e7d788a8e6ff76a6127ce0842b93ed63841fd2ab" exitCode=0
Nov 03 12:44:26.108333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.108203    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh" event={"ID":"45d4b012-69a9-4698-8c37-b70d5c0e6296","Type":"ContainerDied","Data":"e605645282ee08507ce5d076e7d788a8e6ff76a6127ce0842b93ed63841fd2ab"}
Nov 03 12:44:26.108333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.108246    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh" event={"ID":"45d4b012-69a9-4698-8c37-b70d5c0e6296","Type":"ContainerDied","Data":"f4cb6e51bf14ee1e95a27f6f0fc2bbd3589e7ebd69c9572a922776e56a0a5b82"}
Nov 03 12:44:26.108333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.108260    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cb6e51bf14ee1e95a27f6f0fc2bbd3589e7ebd69c9572a922776e56a0a5b82"
Nov 03 12:44:26.130959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.130931    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:44:26.258152 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258117    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache\") pod \"45d4b012-69a9-4698-8c37-b70d5c0e6296\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") "
Nov 03 12:44:26.258363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258302    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn\" (UniqueName: \"kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn\") pod \"45d4b012-69a9-4698-8c37-b70d5c0e6296\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") "
Nov 03 12:44:26.258447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258366    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir\") pod \"45d4b012-69a9-4698-8c37-b70d5c0e6296\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") "
Nov 03 12:44:26.258447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258397    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm\") pod \"45d4b012-69a9-4698-8c37-b70d5c0e6296\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") "
Nov 03 12:44:26.258447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258421    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache" pod "45d4b012-69a9-4698-8c37-b70d5c0e6296" (UID: "45d4b012-69a9-4698-8c37-b70d5c0e6296"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:26.258634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258508    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs\") pod \"45d4b012-69a9-4698-8c37-b70d5c0e6296\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") "
Nov 03 12:44:26.258634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258576    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home\") pod \"45d4b012-69a9-4698-8c37-b70d5c0e6296\" (UID: \"45d4b012-69a9-4698-8c37-b70d5c0e6296\") "
Nov 03 12:44:26.258634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258622    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir" pod "45d4b012-69a9-4698-8c37-b70d5c0e6296" (UID: "45d4b012-69a9-4698-8c37-b70d5c0e6296"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:26.258925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.258891    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home" pod "45d4b012-69a9-4698-8c37-b70d5c0e6296" (UID: "45d4b012-69a9-4698-8c37-b70d5c0e6296"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:26.259229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.259182    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:26.259229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.259202    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:26.259229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.259217    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:26.260774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.260728    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm" pod "45d4b012-69a9-4698-8c37-b70d5c0e6296" (UID: "45d4b012-69a9-4698-8c37-b70d5c0e6296"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:26.260960 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.260938    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn" pod "45d4b012-69a9-4698-8c37-b70d5c0e6296" (UID: "45d4b012-69a9-4698-8c37-b70d5c0e6296"). InnerVolumeSpecName "kube-api-access-pl7rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:44:26.261275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.261250    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs" pod "45d4b012-69a9-4698-8c37-b70d5c0e6296" (UID: "45d4b012-69a9-4698-8c37-b70d5c0e6296"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:44:26.360914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.360837    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7rn\" (UniqueName: \"kubernetes.io/projected/45d4b012-69a9-4698-8c37-b70d5c0e6296-kube-api-access-pl7rn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:26.360914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.360864    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b012-69a9-4698-8c37-b70d5c0e6296-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:26.360914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:26.360885    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b012-69a9-4698-8c37-b70d5c0e6296-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:27.114789 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:27.114741    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"
Nov 03 12:44:27.267392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:27.267358    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"]
Nov 03 12:44:27.313200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:27.313166    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/auth-test-model-cacb84-kserve-74554d4645-kmhxh"]
Nov 03 12:44:28.575022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:28.574930    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d4b012-69a9-4698-8c37-b70d5c0e6296" path="/var/lib/kubelet/pods/45d4b012-69a9-4698-8c37-b70d5c0e6296/volumes"
Nov 03 12:44:33.185060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.185026    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"]
Nov 03 12:44:33.185814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.185784    4546 cpu_manager.go:414] "containerMap: removing container" podUID="45d4b012-69a9-4698-8c37-b70d5c0e6296" containerName="main"
Nov 03 12:44:33.185814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.185808    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d4b012-69a9-4698-8c37-b70d5c0e6296" containerName="main"
Nov 03 12:44:33.186007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.185863    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0f9e2d3f-7447-45a1-a9d7-2495e535cd13" containerName="revoke-keys"
Nov 03 12:44:33.186007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.185872    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9e2d3f-7447-45a1-a9d7-2495e535cd13" containerName="revoke-keys"
Nov 03 12:44:33.186007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.186004    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9e2d3f-7447-45a1-a9d7-2495e535cd13" containerName="revoke-keys"
Nov 03 12:44:33.186125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.186020    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d4b012-69a9-4698-8c37-b70d5c0e6296" containerName="main"
Nov 03 12:44:33.189691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.189674    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.242696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.242666    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4fpc\" (UniqueName: \"kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc\") pod \"payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.242949 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.242922    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"]
Nov 03 12:44:33.243124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.243075    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.285328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.285274    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bebc6d\""
Nov 03 12:44:33.285767 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.285727    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bebc6d-dockercfg-m6ntj\""
Nov 03 12:44:33.344861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.344816    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.345107 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.345053    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fpc\" (UniqueName: \"kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc\") pod \"payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.346414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.346387    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.349055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.349032    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"]
Nov 03 12:44:33.354510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.354489    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.398348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.398311    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fpc\" (UniqueName: \"kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc\") pod \"payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.408685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.408655    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"]
Nov 03 12:44:33.446270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.446183    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskls\" (UniqueName: \"kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls\") pod \"payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.446455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.446378    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.502911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.502880    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:33.547822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.547787    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskls\" (UniqueName: \"kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls\") pod \"payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.548022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.547925    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.548656 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.548633    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.600905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.600869    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskls\" (UniqueName: \"kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls\") pod \"payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.671848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.671814    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:33.881658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:33.881634    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"]
Nov 03 12:44:33.882354 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:44:33.882332    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33462d86_9e7c_4129_8559_d0a32085014b.slice/crio-8cf93db0f08bf7c95b569ef1c08667c081bb9ccc8250256e935e25c38b8a2933 WatchSource:0}: Error finding container 8cf93db0f08bf7c95b569ef1c08667c081bb9ccc8250256e935e25c38b8a2933: Status 404 returned error can't find the container with id 8cf93db0f08bf7c95b569ef1c08667c081bb9ccc8250256e935e25c38b8a2933
Nov 03 12:44:34.054744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.054707    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"]
Nov 03 12:44:34.161123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.161085    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl" event={"ID":"2ee2bf08-7529-4e98-8de0-e985aa54ffcb","Type":"ContainerStarted","Data":"cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176"}
Nov 03 12:44:34.161329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.161148    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl" event={"ID":"2ee2bf08-7529-4e98-8de0-e985aa54ffcb","Type":"ContainerStarted","Data":"0b4b02f68a7d617604f302bb788d25d3b187de1c1413e04a557fbfc0c9239d99"}
Nov 03 12:44:34.161329 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.161263    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:34.164190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.164165    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2" event={"ID":"33462d86-9e7c-4129-8559-d0a32085014b","Type":"ContainerStarted","Data":"930d97070cbf71bf7e7fc6164ce8d89c5dda215c2cc1396251b9066802c8a1e1"}
Nov 03 12:44:34.164349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.164202    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2" event={"ID":"33462d86-9e7c-4129-8559-d0a32085014b","Type":"ContainerStarted","Data":"8cf93db0f08bf7c95b569ef1c08667c081bb9ccc8250256e935e25c38b8a2933"}
Nov 03 12:44:34.164349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.164285    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:34.268312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.268265    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl" podStartSLOduration=1.268249333 podStartE2EDuration="1.268249333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:34.265141458 +0000 UTC m=+3215.836725388" watchObservedRunningTime="2026-11-03 12:44:34.268249333 +0000 UTC m=+3215.839833244"
Nov 03 12:44:34.373095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:34.373005    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2" podStartSLOduration=1.372990951 podStartE2EDuration="1.372990951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:34.372578892 +0000 UTC m=+3215.944162816" watchObservedRunningTime="2026-11-03 12:44:34.372990951 +0000 UTC m=+3215.944574883"
Nov 03 12:44:37.652313 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:44:37.652274    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:44:41.015784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.015715    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m" podUID="fadae778-5912-4c36-a576-1d31e79af6d7" containerName="main" containerID="cri-o://24e3a84915c00b9f375786d540c4cb5552ef69c51fea440345467e8562b45580" gracePeriod=45
Nov 03 12:44:41.211184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.211156    4546 generic.go:356] "Generic (PLEG): container finished" podID="fadae778-5912-4c36-a576-1d31e79af6d7" containerID="24e3a84915c00b9f375786d540c4cb5552ef69c51fea440345467e8562b45580" exitCode=0
Nov 03 12:44:41.211370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.211252    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m" event={"ID":"fadae778-5912-4c36-a576-1d31e79af6d7","Type":"ContainerDied","Data":"24e3a84915c00b9f375786d540c4cb5552ef69c51fea440345467e8562b45580"}
Nov 03 12:44:41.292849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.292824    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:44:41.354542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354518    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir\") pod \"fadae778-5912-4c36-a576-1d31e79af6d7\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") "
Nov 03 12:44:41.354679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354554    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8\" (UniqueName: \"kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8\") pod \"fadae778-5912-4c36-a576-1d31e79af6d7\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") "
Nov 03 12:44:41.354679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354596    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm\") pod \"fadae778-5912-4c36-a576-1d31e79af6d7\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") "
Nov 03 12:44:41.354679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354621    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs\") pod \"fadae778-5912-4c36-a576-1d31e79af6d7\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") "
Nov 03 12:44:41.354679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354651    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home\") pod \"fadae778-5912-4c36-a576-1d31e79af6d7\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") "
Nov 03 12:44:41.354956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354697    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache\") pod \"fadae778-5912-4c36-a576-1d31e79af6d7\" (UID: \"fadae778-5912-4c36-a576-1d31e79af6d7\") "
Nov 03 12:44:41.354956 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.354797    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir" pod "fadae778-5912-4c36-a576-1d31e79af6d7" (UID: "fadae778-5912-4c36-a576-1d31e79af6d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:41.355086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.355013    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache" pod "fadae778-5912-4c36-a576-1d31e79af6d7" (UID: "fadae778-5912-4c36-a576-1d31e79af6d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:41.355086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.355031    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home" pod "fadae778-5912-4c36-a576-1d31e79af6d7" (UID: "fadae778-5912-4c36-a576-1d31e79af6d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:41.355379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.355328    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:41.355379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.355348    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:41.355379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.355378    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:41.356995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.356973    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm" pod "fadae778-5912-4c36-a576-1d31e79af6d7" (UID: "fadae778-5912-4c36-a576-1d31e79af6d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:44:41.357082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.357042    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs" pod "fadae778-5912-4c36-a576-1d31e79af6d7" (UID: "fadae778-5912-4c36-a576-1d31e79af6d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:44:41.357082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.357047    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8" pod "fadae778-5912-4c36-a576-1d31e79af6d7" (UID: "fadae778-5912-4c36-a576-1d31e79af6d7"). InnerVolumeSpecName "kube-api-access-wh5t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:44:41.456177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.456151    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5t8\" (UniqueName: \"kubernetes.io/projected/fadae778-5912-4c36-a576-1d31e79af6d7-kube-api-access-wh5t8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:41.456177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.456174    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fadae778-5912-4c36-a576-1d31e79af6d7-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:41.456177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:41.456184    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fadae778-5912-4c36-a576-1d31e79af6d7-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:44:42.219087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:42.219052    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"
Nov 03 12:44:42.219087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:42.219075    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m" event={"ID":"fadae778-5912-4c36-a576-1d31e79af6d7","Type":"ContainerDied","Data":"3ba150c9449d160196bf4632fccbeae296bff559479ad61062e82b66237dff7b"}
Nov 03 12:44:42.219578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:42.219126    4546 scope.go:122] "RemoveContainer" containerID="24e3a84915c00b9f375786d540c4cb5552ef69c51fea440345467e8562b45580"
Nov 03 12:44:42.372297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:42.372265    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"]
Nov 03 12:44:42.418190 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:42.418161    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/auth-test-model-8570a0-kserve-6784ff67c7-tpv9m"]
Nov 03 12:44:42.573427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:42.573343    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadae778-5912-4c36-a576-1d31e79af6d7" path="/var/lib/kubelet/pods/fadae778-5912-4c36-a576-1d31e79af6d7/volumes"
Nov 03 12:44:43.172076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.172044    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"]
Nov 03 12:44:43.172735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.172713    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fadae778-5912-4c36-a576-1d31e79af6d7" containerName="main"
Nov 03 12:44:43.172841 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.172738    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadae778-5912-4c36-a576-1d31e79af6d7" containerName="main"
Nov 03 12:44:43.173002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.172985    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadae778-5912-4c36-a576-1d31e79af6d7" containerName="main"
Nov 03 12:44:43.178233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.178212    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.231710 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.231676    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"]
Nov 03 12:44:43.273255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.273220    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-fd7881-openshift-default-dockercfg-s6d5m\""
Nov 03 12:44:43.279695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.279665    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.279918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.279705    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjfnr\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.279918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.279736    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.279918 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.279891    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.280079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.279968    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.280189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.280171    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.280249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.280235    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.280330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.280310    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.280390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.280354    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.382824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.382781    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383082    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383129    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383182    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383214    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjfnr\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383254    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383284    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383317    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383480    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.383952 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.383932    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.384458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.384429    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.384675 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.384660    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.385229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.385203    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.385337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.385274    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.390806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.388410    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.393019 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.392990    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.438631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.438551    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.438812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.438645    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjfnr\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr\") pod \"e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") " pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.491598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.491566    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:43.913611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.913577    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"]
Nov 03 12:44:43.914323 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:44:43.914298    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4c38e5_fc8e_40e4_ad27_bf4b3e260342.slice/crio-ece7771fec5c6ccd13379fe698d0e00b7b4f4b23c9a8e66ac58c06aaa8e6e08e WatchSource:0}: Error finding container ece7771fec5c6ccd13379fe698d0e00b7b4f4b23c9a8e66ac58c06aaa8e6e08e: Status 404 returned error can't find the container with id ece7771fec5c6ccd13379fe698d0e00b7b4f4b23c9a8e66ac58c06aaa8e6e08e
Nov 03 12:44:43.916864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.916837    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:44:43.916941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.916897    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:44:43.916941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:43.916934    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:44:44.236655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:44.236611    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" event={"ID":"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342","Type":"ContainerStarted","Data":"9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"}
Nov 03 12:44:44.237095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:44.236679    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" event={"ID":"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342","Type":"ContainerStarted","Data":"ece7771fec5c6ccd13379fe698d0e00b7b4f4b23c9a8e66ac58c06aaa8e6e08e"}
Nov 03 12:44:44.345896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:44.345836    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podStartSLOduration=1.345820453 podStartE2EDuration="1.345820453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:44.343524783 +0000 UTC m=+3225.915108706" watchObservedRunningTime="2026-11-03 12:44:44.345820453 +0000 UTC m=+3225.917404365"
Nov 03 12:44:44.491971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:44.491888    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:44.497879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:44.497846    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:45.184741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:45.181363    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:44:45.187940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:45.187915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:44:45.243341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:45.243306    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:45.244185 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:45.244166    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:44:52.636873 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.636838    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"]
Nov 03 12:44:52.641068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.641048    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.716770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.712049    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"]
Nov 03 12:44:52.716770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.713145    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxh9\" (UniqueName: \"kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9\") pod \"payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.716770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.713188    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.764775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.746125    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fd7881-dockercfg-8pzbt\""
Nov 03 12:44:52.764775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.746907    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fd7881\""
Nov 03 12:44:52.835858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.835817    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxh9\" (UniqueName: \"kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9\") pod \"payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.835858 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.835859    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.836652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.836632    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.889652 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.889561    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxh9\" (UniqueName: \"kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9\") pod \"payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:52.955799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:52.955758    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:53.343897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.343864    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"]
Nov 03 12:44:53.736011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.735976    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"]
Nov 03 12:44:53.740229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.740209    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.788423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.788396    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-52ab15-openshift-default-dockercfg-dxwhd\""
Nov 03 12:44:53.839560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.839528    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"]
Nov 03 12:44:53.851676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.851640    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.851855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.851787    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.851855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.851847    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.851962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.851887    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.852082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.852050    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqqr\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.852159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.852144    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.852196 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.852176    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.852242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.852219    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.852346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.852326    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.953720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.953685    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.953720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.953723    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.953790    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.953833    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.953978    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954041    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954197 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954157    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqqr\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954199    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954214    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954266    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954269    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954323    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954488    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.954676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.954560    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.956199 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.956177    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:53.956644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:53.956627    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:54.025822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.006970    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:54.025822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.007094    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqqr\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr\") pod \"e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") " pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:54.054396 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.054369    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:54.300355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.300266    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv" event={"ID":"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c","Type":"ContainerStarted","Data":"f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3"}
Nov 03 12:44:54.300355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.300322    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv" event={"ID":"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c","Type":"ContainerStarted","Data":"4269e3f829eeced0df491b2dcb7697082f0edd2ba80e14326d0e419619aeb05f"}
Nov 03 12:44:54.300355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.300345    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:44:54.413390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.413326    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv" podStartSLOduration=2.413307916 podStartE2EDuration="2.413307916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:54.403975603 +0000 UTC m=+3235.975559514" watchObservedRunningTime="2026-11-03 12:44:54.413307916 +0000 UTC m=+3235.984891830"
Nov 03 12:44:54.548868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.548841    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"]
Nov 03 12:44:54.551603 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:44:54.551542    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbb5ce2_9e39_42c5_b0c4_ee64be463a2d.slice/crio-57af53232add2fcc344fa66e3030d90496cc44c819549e51316c11f2d4286b69 WatchSource:0}: Error finding container 57af53232add2fcc344fa66e3030d90496cc44c819549e51316c11f2d4286b69: Status 404 returned error can't find the container with id 57af53232add2fcc344fa66e3030d90496cc44c819549e51316c11f2d4286b69
Nov 03 12:44:54.554540 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.554505    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:44:54.554624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.554582    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:44:54.554624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:54.554611    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:44:55.334350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:55.334316    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" event={"ID":"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d","Type":"ContainerStarted","Data":"62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f"}
Nov 03 12:44:55.334788 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:55.334374    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" event={"ID":"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d","Type":"ContainerStarted","Data":"57af53232add2fcc344fa66e3030d90496cc44c819549e51316c11f2d4286b69"}
Nov 03 12:44:55.492684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:55.492625    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podStartSLOduration=2.492609412 podStartE2EDuration="2.492609412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:55.490432202 +0000 UTC m=+3237.062016114" watchObservedRunningTime="2026-11-03 12:44:55.492609412 +0000 UTC m=+3237.064193327"
Nov 03 12:44:56.055691 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:56.055541    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:44:56.057033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:56.056894    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:44:56.057033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:56.056963    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:44:57.057779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:57.057162    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:44:57.057779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:57.057227    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:44:58.054650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:58.054617    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:44:58.054850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:58.054668    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:44:59.055334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:59.055294    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:44:59.055724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:44:59.055368    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:45:00.055590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.055544    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:45:00.056030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.055619    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:45:00.158433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.158395    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"]
Nov 03 12:45:00.163423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.163391    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"]
Nov 03 12:45:00.163621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.163534    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.168140 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.168114    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"]
Nov 03 12:45:00.168283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.168234    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.172418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.172396    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.203104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.203067    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"]
Nov 03 12:45:00.233776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.215127    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"]
Nov 03 12:45:00.233776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.231987    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.233776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.233518    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"]
Nov 03 12:45:00.234076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.233975    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.238607 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.238585    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"]
Nov 03 12:45:00.238800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.238683    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.244026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.244003    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"]
Nov 03 12:45:00.244275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.244254    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.247048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.246009    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:45:00.247048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.246628    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:45:00.247048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.246742    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:45:00.248549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.248528    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"]
Nov 03 12:45:00.250212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.250192    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.253300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.253280    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"]
Nov 03 12:45:00.253512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.253465    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.260899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.260866    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qnk\" (UniqueName: \"kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.261024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.260908    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.261024 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261007    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.261153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261084    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzd97\" (UniqueName: \"kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.261153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261127    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drf5w\" (UniqueName: \"kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.261264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261158    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.261264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261196    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.261264 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261245    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895165-gr8jz\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.261426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261320    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.261426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261366    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxvg\" (UniqueName: \"kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.261426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261411    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9w4\" (UniqueName: \"kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.261598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261486    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnwg\" (UniqueName: \"kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg\") pod \"maas-api-key-cleanup-29895165-gr8jz\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.261598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261535    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.261598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261572    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.261782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261676    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2khd\" (UniqueName: \"kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.261782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.261713    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwppg\" (UniqueName: \"kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.262050 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.262029    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"]
Nov 03 12:45:00.262234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.262220    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.265799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.265778    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"]
Nov 03 12:45:00.274786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.271478    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"]
Nov 03 12:45:00.305033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.304998    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"]
Nov 03 12:45:00.358825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.357525    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"]
Nov 03 12:45:00.362672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.361145    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"]
Nov 03 12:45:00.362849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.362823    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxvg\" (UniqueName: \"kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.362919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.362873    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbpg2\" (UniqueName: \"kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.362984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.362926    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"]
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363264    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9w4\" (UniqueName: \"kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363315    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnwg\" (UniqueName: \"kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg\") pod \"maas-api-key-cleanup-29895165-gr8jz\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363354    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363382    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363464    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2khd\" (UniqueName: \"kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363494    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwppg\" (UniqueName: \"kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363559    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363613    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qnk\" (UniqueName: \"kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363638    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363664    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.363772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363710    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363853    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzd97\" (UniqueName: \"kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363897    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drf5w\" (UniqueName: \"kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363933    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85d6b\" (UniqueName: \"kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363963    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.363996    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.364035    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895165-gr8jz\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.364446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.364111    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.366510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.365249    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.366510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.365598    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.366694 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.366575    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895165-gr8jz\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.366948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.366920    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.367038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.366999    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.367487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.367459    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.368040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.368019    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.368623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.368602    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.372025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.372004    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"]
Nov 03 12:45:00.380731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.374833    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"]
Nov 03 12:45:00.443927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.443735    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"]
Nov 03 12:45:00.449487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.449450    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9w4\" (UniqueName: \"kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.449712 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.449660    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxvg\" (UniqueName: \"kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.450315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.450262    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2khd\" (UniqueName: \"kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.450449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.450426    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnwg\" (UniqueName: \"kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg\") pod \"maas-api-key-cleanup-29895165-gr8jz\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.450641 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.450461    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzd97\" (UniqueName: \"kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.451284 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.451263    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qnk\" (UniqueName: \"kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.452454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.452438    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwppg\" (UniqueName: \"kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.454172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.454149    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drf5w\" (UniqueName: \"kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.466471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.465615    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.466471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.465677    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.466471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.465795    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85d6b\" (UniqueName: \"kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.466471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.465891    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbpg2\" (UniqueName: \"kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.467262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.466991    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.467650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.467624    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.485900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.485877    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:00.497359 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.497335    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:00.504279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.504253    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:00.552420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.552375    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:00.554151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.554118    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85d6b\" (UniqueName: \"kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:00.557821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.554532    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbpg2\" (UniqueName: \"kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.568697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.568656    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:00.597914 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.597055    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:00.603422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.602546    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:00.687072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.686854    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:00.695028 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.694843    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:00.700151 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:00.700069    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:01.055620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.055263    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:45:01.055620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.055310    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:45:01.136151 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:45:01.136119    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac85d3a5_4860_47bf_8d59_840673a22eb9.slice/crio-82e8fefddbafbccb4577ace277288eecb46f5e2937adaca833645d3e232a27e2 WatchSource:0}: Error finding container 82e8fefddbafbccb4577ace277288eecb46f5e2937adaca833645d3e232a27e2: Status 404 returned error can't find the container with id 82e8fefddbafbccb4577ace277288eecb46f5e2937adaca833645d3e232a27e2
Nov 03 12:45:01.137715 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.137650    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"]
Nov 03 12:45:01.174451 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.174427    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"]
Nov 03 12:45:01.187483 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:45:01.187443    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5698d36_32e3_43a2_bcba_c30bbadf53c2.slice/crio-e5d783dd6e94044031bca831b165b778ca71754936bc14656d6f6f46d2ad107d WatchSource:0}: Error finding container e5d783dd6e94044031bca831b165b778ca71754936bc14656d6f6f46d2ad107d: Status 404 returned error can't find the container with id e5d783dd6e94044031bca831b165b778ca71754936bc14656d6f6f46d2ad107d
Nov 03 12:45:01.286261 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.286233    4546 scope.go:122] "RemoveContainer" containerID="e605645282ee08507ce5d076e7d788a8e6ff76a6127ce0842b93ed63841fd2ab"
Nov 03 12:45:01.344787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.344735    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"]
Nov 03 12:45:01.345174 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:45:01.345151    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e58475d_bb7f_4806_8009_6a818555213b.slice/crio-73f3c4c94d1cdad40198e01c18daa67b58bdf842efe0de74117e8bd6f6e2ec89 WatchSource:0}: Error finding container 73f3c4c94d1cdad40198e01c18daa67b58bdf842efe0de74117e8bd6f6e2ec89: Status 404 returned error can't find the container with id 73f3c4c94d1cdad40198e01c18daa67b58bdf842efe0de74117e8bd6f6e2ec89
Nov 03 12:45:01.396265 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.396197    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerStarted","Data":"e5d783dd6e94044031bca831b165b778ca71754936bc14656d6f6f46d2ad107d"}
Nov 03 12:45:01.397538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.397516    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerStarted","Data":"82e8fefddbafbccb4577ace277288eecb46f5e2937adaca833645d3e232a27e2"}
Nov 03 12:45:01.399233 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.399206    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerStarted","Data":"73f3c4c94d1cdad40198e01c18daa67b58bdf842efe0de74117e8bd6f6e2ec89"}
Nov 03 12:45:01.406175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.403396    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"]
Nov 03 12:45:01.406175 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:45:01.404333    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1720a598_042b_4683_b366_e760359b5826.slice/crio-7a16ebe5cf2b14e44d919f9696684482cb8880e0aa75c3a2e11c588c262dd643 WatchSource:0}: Error finding container 7a16ebe5cf2b14e44d919f9696684482cb8880e0aa75c3a2e11c588c262dd643: Status 404 returned error can't find the container with id 7a16ebe5cf2b14e44d919f9696684482cb8880e0aa75c3a2e11c588c262dd643
Nov 03 12:45:01.416141 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.416108    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"]
Nov 03 12:45:01.417888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.417861    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"]
Nov 03 12:45:01.445911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.445811    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"]
Nov 03 12:45:01.458815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.457306    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"]
Nov 03 12:45:01.544278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.544242    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"]
Nov 03 12:45:01.544710 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:45:01.544686    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce4bff6_b99d_418f_a433_f851ab5a981c.slice/crio-c27cc2d6e1c73c0d5ee07c449c6cb3828893b3fc0b75a1d03fd1f36978d9359d WatchSource:0}: Error finding container c27cc2d6e1c73c0d5ee07c449c6cb3828893b3fc0b75a1d03fd1f36978d9359d: Status 404 returned error can't find the container with id c27cc2d6e1c73c0d5ee07c449c6cb3828893b3fc0b75a1d03fd1f36978d9359d
Nov 03 12:45:01.575306 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:01.575278    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"]
Nov 03 12:45:01.578619 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:45:01.578563    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce118ef_6431_467c_8b57_c7abfa319fea.slice/crio-20a4536fb1ec8484cef7f5e1967567efc382990d3d6a306ec847713a9c78338f WatchSource:0}: Error finding container 20a4536fb1ec8484cef7f5e1967567efc382990d3d6a306ec847713a9c78338f: Status 404 returned error can't find the container with id 20a4536fb1ec8484cef7f5e1967567efc382990d3d6a306ec847713a9c78338f
Nov 03 12:45:02.057516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.056197    4546 patch_prober.go:29] interesting pod/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 12:45:02.057516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.056247    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 12:45:02.414804 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.414765    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerStarted","Data":"0a8f5fbb8696ab768dece43876004b78bba8620673af546c80dc60dd6cf5c6c2"}
Nov 03 12:45:02.414945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.414839    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerStarted","Data":"f5575a21a407689f519f19db2288cdd02fc2d0961d8629dcdb78ec69f4a9a06c"}
Nov 03 12:45:02.419311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.419273    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerStarted","Data":"3c444d909543e8013aa9328093f9c357e7a4d72bd01cd4aab72cd00a1ccd09cf"}
Nov 03 12:45:02.421506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.421477    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerStarted","Data":"13f91e836ee88828914047e339e51cc794d06a4355db7e1ad70e207b5e930ce5"}
Nov 03 12:45:02.421622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.421521    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerStarted","Data":"dbcb31576a75f0d0cd11ff4a20f7aef5a1230818617a818da19f5f94194faa0f"}
Nov 03 12:45:02.423388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.423361    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerStarted","Data":"9e7920e1291ee9f037e74203d178404414e3b03965649b49376fa4044ec4314f"}
Nov 03 12:45:02.423482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.423409    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerStarted","Data":"8c69040cee1a51881f20b8e771753fa1fcdb55fca541c812e9653e0ffc591249"}
Nov 03 12:45:02.425055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.425024    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerStarted","Data":"8a5ba8b567e69edbf8c073ec3cb40c580a039cfbabb33059b9942f9da1176fa1"}
Nov 03 12:45:02.425184 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.425068    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerStarted","Data":"20a4536fb1ec8484cef7f5e1967567efc382990d3d6a306ec847713a9c78338f"}
Nov 03 12:45:02.426782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.426733    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerStarted","Data":"936e732a7b71cc9a397ac2f6042c5136ad33c3c19372e3ace667a21dc73fe363"}
Nov 03 12:45:02.426893 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.426792    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerStarted","Data":"c27cc2d6e1c73c0d5ee07c449c6cb3828893b3fc0b75a1d03fd1f36978d9359d"}
Nov 03 12:45:02.428639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.428612    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerStarted","Data":"5dd9c6f94e28d28bbb7a2a46bb278fe62061ef2bf7e10155ac078469cc5af11c"}
Nov 03 12:45:02.428930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.428894    4546 scope.go:122] "RemoveContainer" containerID="5dd9c6f94e28d28bbb7a2a46bb278fe62061ef2bf7e10155ac078469cc5af11c"
Nov 03 12:45:02.430901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.430878    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerStarted","Data":"1882010a7150910753f4d7f2e77fcecc95184abc14df8b2843af668a088f0201"}
Nov 03 12:45:02.431098 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.431077    4546 scope.go:122] "RemoveContainer" containerID="1882010a7150910753f4d7f2e77fcecc95184abc14df8b2843af668a088f0201"
Nov 03 12:45:02.433391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.433367    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerStarted","Data":"d1a50478a077928814969067191d46bcef0ca3649d050ad9f8ce4dfb982e7a1a"}
Nov 03 12:45:02.433493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.433411    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerStarted","Data":"a49c141b69b23b6a25c0eea159fcfb7fe9ad4ee0cebfa2d2b0460ce60fe6b0cb"}
Nov 03 12:45:02.435484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.435461    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerStarted","Data":"63c759ff85851255b5eb06f49bfd7e7eaadc019e177a17c2e8a1b4f8fc4ca495"}
Nov 03 12:45:02.435582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.435505    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerStarted","Data":"7a16ebe5cf2b14e44d919f9696684482cb8880e0aa75c3a2e11c588c262dd643"}
Nov 03 12:45:02.526646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.526592    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" podStartSLOduration=2.526577252 podStartE2EDuration="2.526577252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:02.523979949 +0000 UTC m=+3244.095563874" watchObservedRunningTime="2026-11-03 12:45:02.526577252 +0000 UTC m=+3244.098161162"
Nov 03 12:45:02.631965 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.631907    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" podStartSLOduration=2.631894049 podStartE2EDuration="2.631894049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:02.628208909 +0000 UTC m=+3244.199792857" watchObservedRunningTime="2026-11-03 12:45:02.631894049 +0000 UTC m=+3244.203477960"
Nov 03 12:45:02.735384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.735304    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" podStartSLOduration=2.735284821 podStartE2EDuration="2.735284821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:02.732632397 +0000 UTC m=+3244.304216328" watchObservedRunningTime="2026-11-03 12:45:02.735284821 +0000 UTC m=+3244.306868755"
Nov 03 12:45:02.942781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:02.942718    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" podStartSLOduration=2.942704674 podStartE2EDuration="2.942704674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:02.940611975 +0000 UTC m=+3244.512195888" watchObservedRunningTime="2026-11-03 12:45:02.942704674 +0000 UTC m=+3244.514288585"
Nov 03 12:45:03.049718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.049662    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" podStartSLOduration=3.049647684 podStartE2EDuration="3.049647684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.045441761 +0000 UTC m=+3244.617025652" watchObservedRunningTime="2026-11-03 12:45:03.049647684 +0000 UTC m=+3244.621231596"
Nov 03 12:45:03.059251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.059228    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:45:03.059593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.059524    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:45:03.060189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.060173    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:45:03.256631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.256547    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" podStartSLOduration=3.256534001 podStartE2EDuration="3.256534001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.253365413 +0000 UTC m=+3244.824949326" watchObservedRunningTime="2026-11-03 12:45:03.256534001 +0000 UTC m=+3244.828117913"
Nov 03 12:45:03.359702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.359644    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" podStartSLOduration=3.35962935 podStartE2EDuration="3.35962935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.358138484 +0000 UTC m=+3244.929722395" watchObservedRunningTime="2026-11-03 12:45:03.35962935 +0000 UTC m=+3244.931213262"
Nov 03 12:45:03.446607 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.446573    4546 generic.go:356] "Generic (PLEG): container finished" podID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerID="3c444d909543e8013aa9328093f9c357e7a4d72bd01cd4aab72cd00a1ccd09cf" exitCode=60
Nov 03 12:45:03.446833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.446660    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerDied","Data":"3c444d909543e8013aa9328093f9c357e7a4d72bd01cd4aab72cd00a1ccd09cf"}
Nov 03 12:45:03.446998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.446978    4546 scope.go:122] "RemoveContainer" containerID="3c444d909543e8013aa9328093f9c357e7a4d72bd01cd4aab72cd00a1ccd09cf"
Nov 03 12:45:03.448493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.448471    4546 generic.go:356] "Generic (PLEG): container finished" podID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerID="13f91e836ee88828914047e339e51cc794d06a4355db7e1ad70e207b5e930ce5" exitCode=60
Nov 03 12:45:03.448624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.448542    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerDied","Data":"13f91e836ee88828914047e339e51cc794d06a4355db7e1ad70e207b5e930ce5"}
Nov 03 12:45:03.448852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.448834    4546 scope.go:122] "RemoveContainer" containerID="13f91e836ee88828914047e339e51cc794d06a4355db7e1ad70e207b5e930ce5"
Nov 03 12:45:03.450219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.450188    4546 generic.go:356] "Generic (PLEG): container finished" podID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerID="9e7920e1291ee9f037e74203d178404414e3b03965649b49376fa4044ec4314f" exitCode=60
Nov 03 12:45:03.450335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.450236    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerDied","Data":"9e7920e1291ee9f037e74203d178404414e3b03965649b49376fa4044ec4314f"}
Nov 03 12:45:03.450593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.450500    4546 scope.go:122] "RemoveContainer" containerID="9e7920e1291ee9f037e74203d178404414e3b03965649b49376fa4044ec4314f"
Nov 03 12:45:03.452189 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.452167    4546 generic.go:356] "Generic (PLEG): container finished" podID="bce118ef-6431-467c-8b57-c7abfa319fea" containerID="8a5ba8b567e69edbf8c073ec3cb40c580a039cfbabb33059b9942f9da1176fa1" exitCode=60
Nov 03 12:45:03.452303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.452240    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerDied","Data":"8a5ba8b567e69edbf8c073ec3cb40c580a039cfbabb33059b9942f9da1176fa1"}
Nov 03 12:45:03.452479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.452443    4546 scope.go:122] "RemoveContainer" containerID="8a5ba8b567e69edbf8c073ec3cb40c580a039cfbabb33059b9942f9da1176fa1"
Nov 03 12:45:03.453843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.453818    4546 generic.go:356] "Generic (PLEG): container finished" podID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerID="936e732a7b71cc9a397ac2f6042c5136ad33c3c19372e3ace667a21dc73fe363" exitCode=60
Nov 03 12:45:03.453923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.453901    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerDied","Data":"936e732a7b71cc9a397ac2f6042c5136ad33c3c19372e3ace667a21dc73fe363"}
Nov 03 12:45:03.454082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.454068    4546 scope.go:122] "RemoveContainer" containerID="936e732a7b71cc9a397ac2f6042c5136ad33c3c19372e3ace667a21dc73fe363"
Nov 03 12:45:03.455453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.455436    4546 generic.go:356] "Generic (PLEG): container finished" podID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerID="ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9" exitCode=0
Nov 03 12:45:03.455563 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.455500    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" event={"ID":"323fc1aa-b75a-4423-a181-5ffc6b801396","Type":"ContainerDied","Data":"ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"}
Nov 03 12:45:03.455912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.455896    4546 scope.go:122] "RemoveContainer" containerID="ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"
Nov 03 12:45:03.457555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.457465    4546 generic.go:356] "Generic (PLEG): container finished" podID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerID="5dd9c6f94e28d28bbb7a2a46bb278fe62061ef2bf7e10155ac078469cc5af11c" exitCode=60
Nov 03 12:45:03.457555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.457493    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerDied","Data":"5dd9c6f94e28d28bbb7a2a46bb278fe62061ef2bf7e10155ac078469cc5af11c"}
Nov 03 12:45:03.457555 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.457537    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerStarted","Data":"563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260"}
Nov 03 12:45:03.457930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.457807    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:45:03.457930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.457889    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:45:03.458046 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.457933    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:45:03.459567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.459544    4546 generic.go:356] "Generic (PLEG): container finished" podID="3e58475d-bb7f-4806-8009-6a818555213b" containerID="1882010a7150910753f4d7f2e77fcecc95184abc14df8b2843af668a088f0201" exitCode=60
Nov 03 12:45:03.459704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.459629    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerDied","Data":"1882010a7150910753f4d7f2e77fcecc95184abc14df8b2843af668a088f0201"}
Nov 03 12:45:03.459704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.459666    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerStarted","Data":"f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db"}
Nov 03 12:45:03.461280 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.461262    4546 generic.go:356] "Generic (PLEG): container finished" podID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerID="d1a50478a077928814969067191d46bcef0ca3649d050ad9f8ce4dfb982e7a1a" exitCode=60
Nov 03 12:45:03.461398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.461329    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerDied","Data":"d1a50478a077928814969067191d46bcef0ca3649d050ad9f8ce4dfb982e7a1a"}
Nov 03 12:45:03.461520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.461507    4546 scope.go:122] "RemoveContainer" containerID="d1a50478a077928814969067191d46bcef0ca3649d050ad9f8ce4dfb982e7a1a"
Nov 03 12:45:03.463225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.463206    4546 generic.go:356] "Generic (PLEG): container finished" podID="1720a598-042b-4683-b366-e760359b5826" containerID="63c759ff85851255b5eb06f49bfd7e7eaadc019e177a17c2e8a1b4f8fc4ca495" exitCode=60
Nov 03 12:45:03.463333 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.463294    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerDied","Data":"63c759ff85851255b5eb06f49bfd7e7eaadc019e177a17c2e8a1b4f8fc4ca495"}
Nov 03 12:45:03.463479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.463463    4546 scope.go:122] "RemoveContainer" containerID="63c759ff85851255b5eb06f49bfd7e7eaadc019e177a17c2e8a1b4f8fc4ca495"
Nov 03 12:45:03.463562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.463457    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" podStartSLOduration=3.463443942 podStartE2EDuration="3.463443942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.462654427 +0000 UTC m=+3245.034238356" watchObservedRunningTime="2026-11-03 12:45:03.463443942 +0000 UTC m=+3245.035027855"
Nov 03 12:45:03.477936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.476703    4546 generic.go:356] "Generic (PLEG): container finished" podID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerID="0a8f5fbb8696ab768dece43876004b78bba8620673af546c80dc60dd6cf5c6c2" exitCode=60
Nov 03 12:45:03.477936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.477253    4546 scope.go:122] "RemoveContainer" containerID="0a8f5fbb8696ab768dece43876004b78bba8620673af546c80dc60dd6cf5c6c2"
Nov 03 12:45:03.477936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.477296    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerDied","Data":"0a8f5fbb8696ab768dece43876004b78bba8620673af546c80dc60dd6cf5c6c2"}
Nov 03 12:45:03.783621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:03.783497    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" podStartSLOduration=3.783480411 podStartE2EDuration="3.783480411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:03.78098779 +0000 UTC m=+3245.352571704" watchObservedRunningTime="2026-11-03 12:45:03.783480411 +0000 UTC m=+3245.355064323"
Nov 03 12:45:04.119168 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.119121    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" podStartSLOduration=4.119107851 podStartE2EDuration="4.119107851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:04.11860484 +0000 UTC m=+3245.690188763" watchObservedRunningTime="2026-11-03 12:45:04.119107851 +0000 UTC m=+3245.690691763"
Nov 03 12:45:04.485725 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.485687    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" event={"ID":"323fc1aa-b75a-4423-a181-5ffc6b801396","Type":"ContainerStarted","Data":"a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb"}
Nov 03 12:45:04.487502 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.487478    4546 generic.go:356] "Generic (PLEG): container finished" podID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerID="563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260" exitCode=60
Nov 03 12:45:04.487648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.487558    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerDied","Data":"563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260"}
Nov 03 12:45:04.487648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.487620    4546 scope.go:122] "RemoveContainer" containerID="5dd9c6f94e28d28bbb7a2a46bb278fe62061ef2bf7e10155ac078469cc5af11c"
Nov 03 12:45:04.487925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.487909    4546 scope.go:122] "RemoveContainer" containerID="563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260"
Nov 03 12:45:04.488227 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:04.488205    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt_odh-ai-gateway-infra(ac85d3a5-4860-47bf-8d59-840673a22eb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9"
Nov 03 12:45:04.489491 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.489468    4546 generic.go:356] "Generic (PLEG): container finished" podID="3e58475d-bb7f-4806-8009-6a818555213b" containerID="f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db" exitCode=60
Nov 03 12:45:04.489611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.489554    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerDied","Data":"f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db"}
Nov 03 12:45:04.489723 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.489710    4546 scope.go:122] "RemoveContainer" containerID="f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db"
Nov 03 12:45:04.489948 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:04.489926    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh_odh-ai-gateway-infra(3e58475d-bb7f-4806-8009-6a818555213b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" podUID="3e58475d-bb7f-4806-8009-6a818555213b"
Nov 03 12:45:04.491414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.491395    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerStarted","Data":"8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06"}
Nov 03 12:45:04.493423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.493399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerStarted","Data":"0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd"}
Nov 03 12:45:04.495226 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.495206    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerStarted","Data":"dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f"}
Nov 03 12:45:04.497112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.497091    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerStarted","Data":"0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5"}
Nov 03 12:45:04.499020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.499000    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerStarted","Data":"f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d"}
Nov 03 12:45:04.500940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.500911    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerStarted","Data":"25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa"}
Nov 03 12:45:04.502735 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.502718    4546 scope.go:122] "RemoveContainer" containerID="1882010a7150910753f4d7f2e77fcecc95184abc14df8b2843af668a088f0201"
Nov 03 12:45:04.502835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.502816    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerStarted","Data":"73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82"}
Nov 03 12:45:04.504623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:04.504601    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerStarted","Data":"a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f"}
Nov 03 12:45:05.337782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.337718    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:45:05.388063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.388025    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:45:05.388975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.388954    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 03 12:45:05.389091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.388997    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 03 12:45:05.536968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.536937    4546 scope.go:122] "RemoveContainer" containerID="563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260"
Nov 03 12:45:05.537593 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.537560    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt_odh-ai-gateway-infra(ac85d3a5-4860-47bf-8d59-840673a22eb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9"
Nov 03 12:45:05.539011 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.538988    4546 scope.go:122] "RemoveContainer" containerID="f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db"
Nov 03 12:45:05.539296 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.539271    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh_odh-ai-gateway-infra(3e58475d-bb7f-4806-8009-6a818555213b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" podUID="3e58475d-bb7f-4806-8009-6a818555213b"
Nov 03 12:45:05.540837 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.540818    4546 generic.go:356] "Generic (PLEG): container finished" podID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerID="8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06" exitCode=60
Nov 03 12:45:05.540962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.540914    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerDied","Data":"8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06"}
Nov 03 12:45:05.541021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.540983    4546 scope.go:122] "RemoveContainer" containerID="d1a50478a077928814969067191d46bcef0ca3649d050ad9f8ce4dfb982e7a1a"
Nov 03 12:45:05.541255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.541240    4546 scope.go:122] "RemoveContainer" containerID="8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06"
Nov 03 12:45:05.541496 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.541471    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895165-gr8jz_odh-ai-gateway-infra(538e81a7-c04d-4fb3-b1c7-bf0377713c00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00"
Nov 03 12:45:05.546859 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.546834    4546 generic.go:356] "Generic (PLEG): container finished" podID="1720a598-042b-4683-b366-e760359b5826" containerID="0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd" exitCode=60
Nov 03 12:45:05.546969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.546951    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerDied","Data":"0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd"}
Nov 03 12:45:05.547243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.547226    4546 scope.go:122] "RemoveContainer" containerID="0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd"
Nov 03 12:45:05.547470 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.547450    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht_odh-ai-gateway-infra(1720a598-042b-4683-b366-e760359b5826)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" podUID="1720a598-042b-4683-b366-e760359b5826"
Nov 03 12:45:05.549569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.549548    4546 generic.go:356] "Generic (PLEG): container finished" podID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerID="dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f" exitCode=60
Nov 03 12:45:05.549688 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.549612    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerDied","Data":"dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f"}
Nov 03 12:45:05.549857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.549840    4546 scope.go:122] "RemoveContainer" containerID="dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f"
Nov 03 12:45:05.550094 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.550064    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55_odh-ai-gateway-infra(ab114a94-6cfb-4da4-95fe-d196b40e3116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116"
Nov 03 12:45:05.551733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.551714    4546 generic.go:356] "Generic (PLEG): container finished" podID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerID="0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5" exitCode=60
Nov 03 12:45:05.551863 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.551810    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerDied","Data":"0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5"}
Nov 03 12:45:05.552029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.552013    4546 scope.go:122] "RemoveContainer" containerID="0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5"
Nov 03 12:45:05.552279 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.552256    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw_odh-ai-gateway-infra(c5698d36-32e3-43a2-bcba-c30bbadf53c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2"
Nov 03 12:45:05.553961 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.553940    4546 generic.go:356] "Generic (PLEG): container finished" podID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerID="f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d" exitCode=60
Nov 03 12:45:05.554063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.554019    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerDied","Data":"f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d"}
Nov 03 12:45:05.554222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.554201    4546 scope.go:122] "RemoveContainer" containerID="f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d"
Nov 03 12:45:05.554440 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.554422    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p_odh-ai-gateway-infra(889e3dec-ac77-4225-af3d-cd780a6e5012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012"
Nov 03 12:45:05.556080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.556059    4546 generic.go:356] "Generic (PLEG): container finished" podID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerID="25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa" exitCode=60
Nov 03 12:45:05.556181 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.556138    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerDied","Data":"25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa"}
Nov 03 12:45:05.556370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.556352    4546 scope.go:122] "RemoveContainer" containerID="25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa"
Nov 03 12:45:05.556588 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.556569    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94_odh-ai-gateway-infra(6a6f7a41-8107-4616-a8ba-db85a462c0ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea"
Nov 03 12:45:05.558026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.558006    4546 generic.go:356] "Generic (PLEG): container finished" podID="bce118ef-6431-467c-8b57-c7abfa319fea" containerID="73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82" exitCode=60
Nov 03 12:45:05.558105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.558082    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerDied","Data":"73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82"}
Nov 03 12:45:05.558276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.558259    4546 scope.go:122] "RemoveContainer" containerID="73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82"
Nov 03 12:45:05.558490 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.558472    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5_odh-ai-gateway-infra(bce118ef-6431-467c-8b57-c7abfa319fea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" podUID="bce118ef-6431-467c-8b57-c7abfa319fea"
Nov 03 12:45:05.558647 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.558635    4546 scope.go:122] "RemoveContainer" containerID="63c759ff85851255b5eb06f49bfd7e7eaadc019e177a17c2e8a1b4f8fc4ca495"
Nov 03 12:45:05.560052 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.560034    4546 generic.go:356] "Generic (PLEG): container finished" podID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerID="a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f" exitCode=60
Nov 03 12:45:05.560150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.560086    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerDied","Data":"a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f"}
Nov 03 12:45:05.560344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.560328    4546 scope.go:122] "RemoveContainer" containerID="a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f"
Nov 03 12:45:05.560570 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:05.560549    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp_odh-ai-gateway-infra(3ce4bff6-b99d-418f-a433-f851ab5a981c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c"
Nov 03 12:45:05.572018 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.571993    4546 scope.go:122] "RemoveContainer" containerID="0a8f5fbb8696ab768dece43876004b78bba8620673af546c80dc60dd6cf5c6c2"
Nov 03 12:45:05.585228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.585205    4546 scope.go:122] "RemoveContainer" containerID="3c444d909543e8013aa9328093f9c357e7a4d72bd01cd4aab72cd00a1ccd09cf"
Nov 03 12:45:05.600124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.600101    4546 scope.go:122] "RemoveContainer" containerID="13f91e836ee88828914047e339e51cc794d06a4355db7e1ad70e207b5e930ce5"
Nov 03 12:45:05.657974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.657953    4546 scope.go:122] "RemoveContainer" containerID="9e7920e1291ee9f037e74203d178404414e3b03965649b49376fa4044ec4314f"
Nov 03 12:45:05.670958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.670931    4546 scope.go:122] "RemoveContainer" containerID="8a5ba8b567e69edbf8c073ec3cb40c580a039cfbabb33059b9942f9da1176fa1"
Nov 03 12:45:05.684176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:05.684157    4546 scope.go:122] "RemoveContainer" containerID="936e732a7b71cc9a397ac2f6042c5136ad33c3c19372e3ace667a21dc73fe363"
Nov 03 12:45:06.388624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.388590    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 03 12:45:06.389079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.388644    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 03 12:45:06.568799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.568774    4546 scope.go:122] "RemoveContainer" containerID="8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06"
Nov 03 12:45:06.569093 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.569072    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895165-gr8jz_odh-ai-gateway-infra(538e81a7-c04d-4fb3-b1c7-bf0377713c00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00"
Nov 03 12:45:06.570459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.570438    4546 scope.go:122] "RemoveContainer" containerID="0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd"
Nov 03 12:45:06.570739 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.570719    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht_odh-ai-gateway-infra(1720a598-042b-4683-b366-e760359b5826)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" podUID="1720a598-042b-4683-b366-e760359b5826"
Nov 03 12:45:06.572000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.571972    4546 scope.go:122] "RemoveContainer" containerID="dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f"
Nov 03 12:45:06.572247 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.572226    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55_odh-ai-gateway-infra(ab114a94-6cfb-4da4-95fe-d196b40e3116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116"
Nov 03 12:45:06.573580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.573560    4546 scope.go:122] "RemoveContainer" containerID="0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5"
Nov 03 12:45:06.573821 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.573791    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw_odh-ai-gateway-infra(c5698d36-32e3-43a2-bcba-c30bbadf53c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2"
Nov 03 12:45:06.575093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.575076    4546 scope.go:122] "RemoveContainer" containerID="f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d"
Nov 03 12:45:06.575306 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.575287    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p_odh-ai-gateway-infra(889e3dec-ac77-4225-af3d-cd780a6e5012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012"
Nov 03 12:45:06.576577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.576560    4546 scope.go:122] "RemoveContainer" containerID="25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa"
Nov 03 12:45:06.576777 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.576738    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94_odh-ai-gateway-infra(6a6f7a41-8107-4616-a8ba-db85a462c0ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea"
Nov 03 12:45:06.578053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.578036    4546 scope.go:122] "RemoveContainer" containerID="73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82"
Nov 03 12:45:06.578224 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.578206    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5_odh-ai-gateway-infra(bce118ef-6431-467c-8b57-c7abfa319fea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" podUID="bce118ef-6431-467c-8b57-c7abfa319fea"
Nov 03 12:45:06.579374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:06.579359    4546 scope.go:122] "RemoveContainer" containerID="a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f"
Nov 03 12:45:06.579543 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:06.579526    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp_odh-ai-gateway-infra(3ce4bff6-b99d-418f-a433-f851ab5a981c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c"
Nov 03 12:45:07.388604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:07.388572    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 03 12:45:07.388833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:07.388630    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 03 12:45:07.662089 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:07.662005    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:45:08.388515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:08.388482    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 03 12:45:08.388682 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:08.388537    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 03 12:45:09.388878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:09.388845    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 03 12:45:09.389355 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:09.388906    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 03 12:45:10.393628 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:10.393595    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:45:10.394002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:10.393968    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:45:10.394700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:10.394680    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 12:45:14.566337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:14.566304    4546 scope.go:122] "RemoveContainer" containerID="f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db"
Nov 03 12:45:15.565915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.565883    4546 scope.go:122] "RemoveContainer" containerID="dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f"
Nov 03 12:45:15.566121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.566032    4546 scope.go:122] "RemoveContainer" containerID="563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260"
Nov 03 12:45:15.566168 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.566150    4546 scope.go:122] "RemoveContainer" containerID="0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd"
Nov 03 12:45:15.642512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.642488    4546 generic.go:356] "Generic (PLEG): container finished" podID="3e58475d-bb7f-4806-8009-6a818555213b" containerID="dc20c1e4d490ba3ca9fed5cac724e7b10326f4ede60343615c1c37d4bd8590a2" exitCode=60
Nov 03 12:45:15.642936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.642560    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerDied","Data":"dc20c1e4d490ba3ca9fed5cac724e7b10326f4ede60343615c1c37d4bd8590a2"}
Nov 03 12:45:15.642936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.642620    4546 scope.go:122] "RemoveContainer" containerID="f2584ab0df78c3cb02cc8d1f994e015f757a65fd543e8273a881d8e2639126db"
Nov 03 12:45:15.643105 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:15.643086    4546 scope.go:122] "RemoveContainer" containerID="dc20c1e4d490ba3ca9fed5cac724e7b10326f4ede60343615c1c37d4bd8590a2"
Nov 03 12:45:15.644202 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:15.644168    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh_odh-ai-gateway-infra(3e58475d-bb7f-4806-8009-6a818555213b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" podUID="3e58475d-bb7f-4806-8009-6a818555213b"
Nov 03 12:45:16.565700 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.565664    4546 scope.go:122] "RemoveContainer" containerID="73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82"
Nov 03 12:45:16.565933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.565831    4546 scope.go:122] "RemoveContainer" containerID="0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5"
Nov 03 12:45:16.566217 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.566195    4546 scope.go:122] "RemoveContainer" containerID="8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06"
Nov 03 12:45:16.652290 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.652257    4546 generic.go:356] "Generic (PLEG): container finished" podID="1720a598-042b-4683-b366-e760359b5826" containerID="0ffe166051e36d975065f723bfa84a51604cb0ccc0a9dd72b54283e9aacc3cbb" exitCode=60
Nov 03 12:45:16.652714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.652325    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerDied","Data":"0ffe166051e36d975065f723bfa84a51604cb0ccc0a9dd72b54283e9aacc3cbb"}
Nov 03 12:45:16.652714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.652386    4546 scope.go:122] "RemoveContainer" containerID="0d0f60e35788564cf374528275d2f17364a3095aae0db63f416be236eccad0cd"
Nov 03 12:45:16.652892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.652875    4546 scope.go:122] "RemoveContainer" containerID="0ffe166051e36d975065f723bfa84a51604cb0ccc0a9dd72b54283e9aacc3cbb"
Nov 03 12:45:16.653322 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:16.653294    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht_odh-ai-gateway-infra(1720a598-042b-4683-b366-e760359b5826)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" podUID="1720a598-042b-4683-b366-e760359b5826"
Nov 03 12:45:16.654888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.654870    4546 generic.go:356] "Generic (PLEG): container finished" podID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerID="f275c21615e1c5c4eb151bacb99487597332f1a5b49d75592138715a521470bb" exitCode=60
Nov 03 12:45:16.654976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.654931    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerDied","Data":"f275c21615e1c5c4eb151bacb99487597332f1a5b49d75592138715a521470bb"}
Nov 03 12:45:16.655410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.655345    4546 scope.go:122] "RemoveContainer" containerID="f275c21615e1c5c4eb151bacb99487597332f1a5b49d75592138715a521470bb"
Nov 03 12:45:16.655762 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:16.655718    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55_odh-ai-gateway-infra(ab114a94-6cfb-4da4-95fe-d196b40e3116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116"
Nov 03 12:45:16.657132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.657113    4546 generic.go:356] "Generic (PLEG): container finished" podID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerID="c11ec61a13741e449af13e776be924b2bdb61e0868d947842591a41fe9cf8b2b" exitCode=60
Nov 03 12:45:16.657242 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.657190    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerDied","Data":"c11ec61a13741e449af13e776be924b2bdb61e0868d947842591a41fe9cf8b2b"}
Nov 03 12:45:16.657536 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.657517    4546 scope.go:122] "RemoveContainer" containerID="c11ec61a13741e449af13e776be924b2bdb61e0868d947842591a41fe9cf8b2b"
Nov 03 12:45:16.657823 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:16.657799    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt_odh-ai-gateway-infra(ac85d3a5-4860-47bf-8d59-840673a22eb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9"
Nov 03 12:45:16.760639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.760610    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"]
Nov 03 12:45:16.771341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.771313    4546 scope.go:122] "RemoveContainer" containerID="dbab17a9756ad2f36b32f3862ad2ef4819ee4584cd57b41dcef69ff936d7d77f"
Nov 03 12:45:16.787684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.787661    4546 scope.go:122] "RemoveContainer" containerID="563ec794d9156de98a5a22503976b16c0937044a7ade4b0cdeba389548c2d260"
Nov 03 12:45:16.954594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:16.954573    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:17.020592 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.020568    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca\") pod \"3e58475d-bb7f-4806-8009-6a818555213b\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") "
Nov 03 12:45:17.020703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.020636    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97\" (UniqueName: \"kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97\") pod \"3e58475d-bb7f-4806-8009-6a818555213b\" (UID: \"3e58475d-bb7f-4806-8009-6a818555213b\") "
Nov 03 12:45:17.020999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.020954    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca" pod "3e58475d-bb7f-4806-8009-6a818555213b" (UID: "3e58475d-bb7f-4806-8009-6a818555213b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:17.021318 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.021297    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e58475d-bb7f-4806-8009-6a818555213b-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:17.023495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.023467    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97" pod "3e58475d-bb7f-4806-8009-6a818555213b" (UID: "3e58475d-bb7f-4806-8009-6a818555213b"). InnerVolumeSpecName "kube-api-access-zzd97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:17.122195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.122170    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzd97\" (UniqueName: \"kubernetes.io/projected/3e58475d-bb7f-4806-8009-6a818555213b-kube-api-access-zzd97\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:17.566959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.566933    4546 scope.go:122] "RemoveContainer" containerID="25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa"
Nov 03 12:45:17.670897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.670870    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"
Nov 03 12:45:17.671305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.670892    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh" event={"ID":"3e58475d-bb7f-4806-8009-6a818555213b","Type":"ContainerDied","Data":"73f3c4c94d1cdad40198e01c18daa67b58bdf842efe0de74117e8bd6f6e2ec89"}
Nov 03 12:45:17.671305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.670963    4546 scope.go:122] "RemoveContainer" containerID="dc20c1e4d490ba3ca9fed5cac724e7b10326f4ede60343615c1c37d4bd8590a2"
Nov 03 12:45:17.673793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.673770    4546 generic.go:356] "Generic (PLEG): container finished" podID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerID="94908f4f53fa449f3959faec916f6c05db5713ef68cd74f55586e87c7ab89603" exitCode=60
Nov 03 12:45:17.673929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.673845    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerDied","Data":"94908f4f53fa449f3959faec916f6c05db5713ef68cd74f55586e87c7ab89603"}
Nov 03 12:45:17.674328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.674310    4546 scope.go:122] "RemoveContainer" containerID="94908f4f53fa449f3959faec916f6c05db5713ef68cd74f55586e87c7ab89603"
Nov 03 12:45:17.674677 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:17.674656    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895165-gr8jz_odh-ai-gateway-infra(538e81a7-c04d-4fb3-b1c7-bf0377713c00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00"
Nov 03 12:45:17.684698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.684675    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerStarted","Data":"10f808952bcfc217224d8bd8c2012ec20f41484fdbae38d56dec3a727b385389"}
Nov 03 12:45:17.685392 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.685375    4546 scope.go:122] "RemoveContainer" containerID="8256a1a97830eae6eda40710e0ee91c49e79a6d0e0ffdc7ec8c45ca790b01e06"
Nov 03 12:45:17.687180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.687162    4546 generic.go:356] "Generic (PLEG): container finished" podID="bce118ef-6431-467c-8b57-c7abfa319fea" containerID="cae424660fb572f92bf3f7bf1867939a9e8b8314ba70d021021e3b644fed6c0e" exitCode=60
Nov 03 12:45:17.687273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.687215    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerDied","Data":"cae424660fb572f92bf3f7bf1867939a9e8b8314ba70d021021e3b644fed6c0e"}
Nov 03 12:45:17.687521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.687509    4546 scope.go:122] "RemoveContainer" containerID="cae424660fb572f92bf3f7bf1867939a9e8b8314ba70d021021e3b644fed6c0e"
Nov 03 12:45:17.687738 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:17.687722    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5_odh-ai-gateway-infra(bce118ef-6431-467c-8b57-c7abfa319fea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" podUID="bce118ef-6431-467c-8b57-c7abfa319fea"
Nov 03 12:45:17.698310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.698289    4546 scope.go:122] "RemoveContainer" containerID="73ad7915e287122698435bfb8750bb393a27d180f365d394f294d5210660dc82"
Nov 03 12:45:17.776709 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.776655    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"]
Nov 03 12:45:17.875062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.875033    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"]
Nov 03 12:45:17.935533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.935475    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"]
Nov 03 12:45:17.939600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.939577    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d2c7c-29895165-h5ljh"]
Nov 03 12:45:17.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.958116    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:17.978889 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:17.978849    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"]
Nov 03 12:45:18.030409 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.030346    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:18.033678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.033652    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca\") pod \"1720a598-042b-4683-b366-e760359b5826\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") "
Nov 03 12:45:18.033895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.033878    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg\" (UniqueName: \"kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg\") pod \"1720a598-042b-4683-b366-e760359b5826\" (UID: \"1720a598-042b-4683-b366-e760359b5826\") "
Nov 03 12:45:18.033981 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.033962    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca" pod "1720a598-042b-4683-b366-e760359b5826" (UID: "1720a598-042b-4683-b366-e760359b5826"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:18.034326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.034306    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1720a598-042b-4683-b366-e760359b5826-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.036111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.036090    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg" pod "1720a598-042b-4683-b366-e760359b5826" (UID: "1720a598-042b-4683-b366-e760359b5826"). InnerVolumeSpecName "kube-api-access-jwppg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:18.134780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.134733    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca\") pod \"ab114a94-6cfb-4da4-95fe-d196b40e3116\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") "
Nov 03 12:45:18.134939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.134843    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w\" (UniqueName: \"kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w\") pod \"ab114a94-6cfb-4da4-95fe-d196b40e3116\" (UID: \"ab114a94-6cfb-4da4-95fe-d196b40e3116\") "
Nov 03 12:45:18.135056 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.135032    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca" pod "ab114a94-6cfb-4da4-95fe-d196b40e3116" (UID: "ab114a94-6cfb-4da4-95fe-d196b40e3116"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:18.135330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.135310    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwppg\" (UniqueName: \"kubernetes.io/projected/1720a598-042b-4683-b366-e760359b5826-kube-api-access-jwppg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.135485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.135332    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab114a94-6cfb-4da4-95fe-d196b40e3116-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.137143 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.137122    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w" pod "ab114a94-6cfb-4da4-95fe-d196b40e3116" (UID: "ab114a94-6cfb-4da4-95fe-d196b40e3116"). InnerVolumeSpecName "kube-api-access-drf5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:18.140055 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.140038    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:18.236417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.236383    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg\" (UniqueName: \"kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg\") pod \"ac85d3a5-4860-47bf-8d59-840673a22eb9\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") "
Nov 03 12:45:18.236587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.236440    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca\") pod \"ac85d3a5-4860-47bf-8d59-840673a22eb9\" (UID: \"ac85d3a5-4860-47bf-8d59-840673a22eb9\") "
Nov 03 12:45:18.236878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.236855    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca" pod "ac85d3a5-4860-47bf-8d59-840673a22eb9" (UID: "ac85d3a5-4860-47bf-8d59-840673a22eb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:18.237035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.237017    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac85d3a5-4860-47bf-8d59-840673a22eb9-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.237095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.237041    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drf5w\" (UniqueName: \"kubernetes.io/projected/ab114a94-6cfb-4da4-95fe-d196b40e3116-kube-api-access-drf5w\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.238604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.238588    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg" pod "ac85d3a5-4860-47bf-8d59-840673a22eb9" (UID: "ac85d3a5-4860-47bf-8d59-840673a22eb9"). InnerVolumeSpecName "kube-api-access-7rxvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:18.338376 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.338303    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxvg\" (UniqueName: \"kubernetes.io/projected/ac85d3a5-4860-47bf-8d59-840673a22eb9-kube-api-access-7rxvg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:18.573577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.573547    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e58475d-bb7f-4806-8009-6a818555213b" path="/var/lib/kubelet/pods/3e58475d-bb7f-4806-8009-6a818555213b/volumes"
Nov 03 12:45:18.575148 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.575127    4546 scope.go:122] "RemoveContainer" containerID="a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f"
Nov 03 12:45:18.575422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.575399    4546 scope.go:122] "RemoveContainer" containerID="f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d"
Nov 03 12:45:18.698389 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.698365    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"
Nov 03 12:45:18.698968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.698360    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt" event={"ID":"ac85d3a5-4860-47bf-8d59-840673a22eb9","Type":"ContainerDied","Data":"82e8fefddbafbccb4577ace277288eecb46f5e2937adaca833645d3e232a27e2"}
Nov 03 12:45:18.698968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.698519    4546 scope.go:122] "RemoveContainer" containerID="c11ec61a13741e449af13e776be924b2bdb61e0868d947842591a41fe9cf8b2b"
Nov 03 12:45:18.705795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.704847    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"
Nov 03 12:45:18.705795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.704858    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht" event={"ID":"1720a598-042b-4683-b366-e760359b5826","Type":"ContainerDied","Data":"7a16ebe5cf2b14e44d919f9696684482cb8880e0aa75c3a2e11c588c262dd643"}
Nov 03 12:45:18.711781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.707677    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55" event={"ID":"ab114a94-6cfb-4da4-95fe-d196b40e3116","Type":"ContainerDied","Data":"f5575a21a407689f519f19db2288cdd02fc2d0961d8629dcdb78ec69f4a9a06c"}
Nov 03 12:45:18.711781 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.707815    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"
Nov 03 12:45:18.712323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.712306    4546 scope.go:122] "RemoveContainer" containerID="0ffe166051e36d975065f723bfa84a51604cb0ccc0a9dd72b54283e9aacc3cbb"
Nov 03 12:45:18.712498 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.712477    4546 generic.go:356] "Generic (PLEG): container finished" podID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerID="10f808952bcfc217224d8bd8c2012ec20f41484fdbae38d56dec3a727b385389" exitCode=60
Nov 03 12:45:18.712582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.712544    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerDied","Data":"10f808952bcfc217224d8bd8c2012ec20f41484fdbae38d56dec3a727b385389"}
Nov 03 12:45:18.712964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.712948    4546 scope.go:122] "RemoveContainer" containerID="10f808952bcfc217224d8bd8c2012ec20f41484fdbae38d56dec3a727b385389"
Nov 03 12:45:18.713255 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:18.713228    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw_odh-ai-gateway-infra(c5698d36-32e3-43a2-bcba-c30bbadf53c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2"
Nov 03 12:45:18.714969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.714938    4546 generic.go:356] "Generic (PLEG): container finished" podID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerID="5089cef1f1d204290346fd869c15cb8adf8a3e97729d0efccef8b13981b86cf7" exitCode=60
Nov 03 12:45:18.715069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.714976    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerDied","Data":"5089cef1f1d204290346fd869c15cb8adf8a3e97729d0efccef8b13981b86cf7"}
Nov 03 12:45:18.715339 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.715323    4546 scope.go:122] "RemoveContainer" containerID="5089cef1f1d204290346fd869c15cb8adf8a3e97729d0efccef8b13981b86cf7"
Nov 03 12:45:18.715551 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:18.715532    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94_odh-ai-gateway-infra(6a6f7a41-8107-4616-a8ba-db85a462c0ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea"
Nov 03 12:45:18.721823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.721803    4546 scope.go:122] "RemoveContainer" containerID="f275c21615e1c5c4eb151bacb99487597332f1a5b49d75592138715a521470bb"
Nov 03 12:45:18.731692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.731666    4546 scope.go:122] "RemoveContainer" containerID="0c398fde42b80fbedd52e328c105b594f1c320edf39f0c66262b70c2b8b730b5"
Nov 03 12:45:18.744846 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.744822    4546 scope.go:122] "RemoveContainer" containerID="25522c1561b7d2d8be75a23e05ff0f6dde2b5e739e6f6861f146debefbe6f3fa"
Nov 03 12:45:18.840213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:18.840179    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"]
Nov 03 12:45:19.029839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.029814    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:19.052158 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.052133    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"]
Nov 03 12:45:19.058744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.058721    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"]
Nov 03 12:45:19.064888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.064868    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895165-smzht"]
Nov 03 12:45:19.148173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.148148    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg\" (UniqueName: \"kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg\") pod \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") "
Nov 03 12:45:19.148298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.148210    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca\") pod \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\" (UID: \"538e81a7-c04d-4fb3-b1c7-bf0377713c00\") "
Nov 03 12:45:19.148548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.148527    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca" pod "538e81a7-c04d-4fb3-b1c7-bf0377713c00" (UID: "538e81a7-c04d-4fb3-b1c7-bf0377713c00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:19.148697 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.148678    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538e81a7-c04d-4fb3-b1c7-bf0377713c00-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:19.150459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.150442    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg" pod "538e81a7-c04d-4fb3-b1c7-bf0377713c00" (UID: "538e81a7-c04d-4fb3-b1c7-bf0377713c00"). InnerVolumeSpecName "kube-api-access-wdnwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:19.160738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.160708    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"]
Nov 03 12:45:19.168720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.168691    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"]
Nov 03 12:45:19.172664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.172555    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895165-6rd55"]
Nov 03 12:45:19.250031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.249999    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnwg\" (UniqueName: \"kubernetes.io/projected/538e81a7-c04d-4fb3-b1c7-bf0377713c00-kube-api-access-wdnwg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:19.275135 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.275095    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"]
Nov 03 12:45:19.279936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.279908    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895165-cv6xt"]
Nov 03 12:45:19.307210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.307184    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:19.351215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.351186    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2\" (UniqueName: \"kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2\") pod \"bce118ef-6431-467c-8b57-c7abfa319fea\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") "
Nov 03 12:45:19.351380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.351235    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca\") pod \"bce118ef-6431-467c-8b57-c7abfa319fea\" (UID: \"bce118ef-6431-467c-8b57-c7abfa319fea\") "
Nov 03 12:45:19.351580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.351557    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca" pod "bce118ef-6431-467c-8b57-c7abfa319fea" (UID: "bce118ef-6431-467c-8b57-c7abfa319fea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:19.351836 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.351822    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce118ef-6431-467c-8b57-c7abfa319fea-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:19.353435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.353415    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2" pod "bce118ef-6431-467c-8b57-c7abfa319fea" (UID: "bce118ef-6431-467c-8b57-c7abfa319fea"). InnerVolumeSpecName "kube-api-access-rbpg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:19.453470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.453392    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbpg2\" (UniqueName: \"kubernetes.io/projected/bce118ef-6431-467c-8b57-c7abfa319fea-kube-api-access-rbpg2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:19.737449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.737421    4546 generic.go:356] "Generic (PLEG): container finished" podID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerID="ff3822328c4819f8fd92420224c7303e3426ec8d89311fab1debd37718ba58be" exitCode=60
Nov 03 12:45:19.737810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.737492    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerDied","Data":"ff3822328c4819f8fd92420224c7303e3426ec8d89311fab1debd37718ba58be"}
Nov 03 12:45:19.737810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.737577    4546 scope.go:122] "RemoveContainer" containerID="f7cfb48d1c51fe848cf9ebdc29c1c2b5d0047450fdb35a96948af7b39ae4939d"
Nov 03 12:45:19.737980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.737964    4546 scope.go:122] "RemoveContainer" containerID="ff3822328c4819f8fd92420224c7303e3426ec8d89311fab1debd37718ba58be"
Nov 03 12:45:19.738302 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:19.738277    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p_odh-ai-gateway-infra(889e3dec-ac77-4225-af3d-cd780a6e5012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012"
Nov 03 12:45:19.740470 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.740447    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5" event={"ID":"bce118ef-6431-467c-8b57-c7abfa319fea","Type":"ContainerDied","Data":"20a4536fb1ec8484cef7f5e1967567efc382990d3d6a306ec847713a9c78338f"}
Nov 03 12:45:19.740578 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.740456    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"
Nov 03 12:45:19.742518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.742495    4546 generic.go:356] "Generic (PLEG): container finished" podID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerID="8329a9aeb593d43e629812e9b1195aee42c768b9bb75b315ee2e3c3c1b0fe38f" exitCode=60
Nov 03 12:45:19.742635 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.742580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerDied","Data":"8329a9aeb593d43e629812e9b1195aee42c768b9bb75b315ee2e3c3c1b0fe38f"}
Nov 03 12:45:19.743053 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.743032    4546 scope.go:122] "RemoveContainer" containerID="8329a9aeb593d43e629812e9b1195aee42c768b9bb75b315ee2e3c3c1b0fe38f"
Nov 03 12:45:19.743356 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:19.743320    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp_odh-ai-gateway-infra(3ce4bff6-b99d-418f-a433-f851ab5a981c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c"
Nov 03 12:45:19.745404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.745378    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"
Nov 03 12:45:19.745505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.745405    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz" event={"ID":"538e81a7-c04d-4fb3-b1c7-bf0377713c00","Type":"ContainerDied","Data":"a49c141b69b23b6a25c0eea159fcfb7fe9ad4ee0cebfa2d2b0460ce60fe6b0cb"}
Nov 03 12:45:19.775888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.775868    4546 scope.go:122] "RemoveContainer" containerID="cae424660fb572f92bf3f7bf1867939a9e8b8314ba70d021021e3b644fed6c0e"
Nov 03 12:45:19.785695 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.785677    4546 scope.go:122] "RemoveContainer" containerID="a7130a84a191e79bb3c96018623947165bc5ce92417e706675d742fac237d25f"
Nov 03 12:45:19.798967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.798950    4546 scope.go:122] "RemoveContainer" containerID="94908f4f53fa449f3959faec916f6c05db5713ef68cd74f55586e87c7ab89603"
Nov 03 12:45:19.902843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.902819    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:19.947288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.947258    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"]
Nov 03 12:45:19.950719 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.950695    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895165-gr8jz"]
Nov 03 12:45:19.961521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.961451    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk\" (UniqueName: \"kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk\") pod \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") "
Nov 03 12:45:19.961632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.961612    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca\") pod \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\" (UID: \"c5698d36-32e3-43a2-bcba-c30bbadf53c2\") "
Nov 03 12:45:19.961963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.961937    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca" pod "c5698d36-32e3-43a2-bcba-c30bbadf53c2" (UID: "c5698d36-32e3-43a2-bcba-c30bbadf53c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:19.962192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.962174    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5698d36-32e3-43a2-bcba-c30bbadf53c2-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:19.963823 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.963797    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk" pod "c5698d36-32e3-43a2-bcba-c30bbadf53c2" (UID: "c5698d36-32e3-43a2-bcba-c30bbadf53c2"). InnerVolumeSpecName "kube-api-access-l6qnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:19.964464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:19.964449    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"]
Nov 03 12:45:20.063908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.063877    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qnk\" (UniqueName: \"kubernetes.io/projected/c5698d36-32e3-43a2-bcba-c30bbadf53c2-kube-api-access-l6qnk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:20.092506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.092482    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:20.164927 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.164893    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd\" (UniqueName: \"kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd\") pod \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") "
Nov 03 12:45:20.165080 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.165020    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca\") pod \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\" (UID: \"6a6f7a41-8107-4616-a8ba-db85a462c0ea\") "
Nov 03 12:45:20.165356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.165335    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca" pod "6a6f7a41-8107-4616-a8ba-db85a462c0ea" (UID: "6a6f7a41-8107-4616-a8ba-db85a462c0ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:20.165524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.165490    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f7a41-8107-4616-a8ba-db85a462c0ea-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:20.167150 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.167126    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd" pod "6a6f7a41-8107-4616-a8ba-db85a462c0ea" (UID: "6a6f7a41-8107-4616-a8ba-db85a462c0ea"). InnerVolumeSpecName "kube-api-access-c2khd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:20.261161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.261116    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"]
Nov 03 12:45:20.264760 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.264721    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895165-99cj5"]
Nov 03 12:45:20.267395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.267372    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2khd\" (UniqueName: \"kubernetes.io/projected/6a6f7a41-8107-4616-a8ba-db85a462c0ea-kube-api-access-c2khd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:20.575760 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.575669    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1720a598-042b-4683-b366-e760359b5826" path="/var/lib/kubelet/pods/1720a598-042b-4683-b366-e760359b5826/volumes"
Nov 03 12:45:20.576126 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.576111    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" path="/var/lib/kubelet/pods/538e81a7-c04d-4fb3-b1c7-bf0377713c00/volumes"
Nov 03 12:45:20.576476 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.576464    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" path="/var/lib/kubelet/pods/ab114a94-6cfb-4da4-95fe-d196b40e3116/volumes"
Nov 03 12:45:20.577123 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.577105    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" path="/var/lib/kubelet/pods/ac85d3a5-4860-47bf-8d59-840673a22eb9/volumes"
Nov 03 12:45:20.577475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.577461    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" path="/var/lib/kubelet/pods/bce118ef-6431-467c-8b57-c7abfa319fea/volumes"
Nov 03 12:45:20.755854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.755815    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"
Nov 03 12:45:20.755854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.755829    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw" event={"ID":"c5698d36-32e3-43a2-bcba-c30bbadf53c2","Type":"ContainerDied","Data":"e5d783dd6e94044031bca831b165b778ca71754936bc14656d6f6f46d2ad107d"}
Nov 03 12:45:20.756363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.755899    4546 scope.go:122] "RemoveContainer" containerID="10f808952bcfc217224d8bd8c2012ec20f41484fdbae38d56dec3a727b385389"
Nov 03 12:45:20.759936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.759905    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"
Nov 03 12:45:20.760057 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.759902    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94" event={"ID":"6a6f7a41-8107-4616-a8ba-db85a462c0ea","Type":"ContainerDied","Data":"8c69040cee1a51881f20b8e771753fa1fcdb55fca541c812e9653e0ffc591249"}
Nov 03 12:45:20.768825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.768703    4546 scope.go:122] "RemoveContainer" containerID="5089cef1f1d204290346fd869c15cb8adf8a3e97729d0efccef8b13981b86cf7"
Nov 03 12:45:20.909505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.909411    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"]
Nov 03 12:45:20.913611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:20.913580    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cacb84-29895165-p7gxw"]
Nov 03 12:45:21.014040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.013995    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"]
Nov 03 12:45:21.017179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.017156    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895165-h6v94"]
Nov 03 12:45:21.063027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.062999    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"]
Nov 03 12:45:21.167692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.167591    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"]
Nov 03 12:45:21.226423 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.226395    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:21.280180 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.280150    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca\") pod \"889e3dec-ac77-4225-af3d-cd780a6e5012\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") "
Nov 03 12:45:21.280350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.280188    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4\" (UniqueName: \"kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4\") pod \"889e3dec-ac77-4225-af3d-cd780a6e5012\" (UID: \"889e3dec-ac77-4225-af3d-cd780a6e5012\") "
Nov 03 12:45:21.280604 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.280576    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca" pod "889e3dec-ac77-4225-af3d-cd780a6e5012" (UID: "889e3dec-ac77-4225-af3d-cd780a6e5012"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:21.280822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.280800    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/889e3dec-ac77-4225-af3d-cd780a6e5012-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:21.282600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.282580    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4" pod "889e3dec-ac77-4225-af3d-cd780a6e5012" (UID: "889e3dec-ac77-4225-af3d-cd780a6e5012"). InnerVolumeSpecName "kube-api-access-5j9w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:21.336868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.336841    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:21.381258 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.381230    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b\" (UniqueName: \"kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b\") pod \"3ce4bff6-b99d-418f-a433-f851ab5a981c\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") "
Nov 03 12:45:21.381415 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.381273    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca\") pod \"3ce4bff6-b99d-418f-a433-f851ab5a981c\" (UID: \"3ce4bff6-b99d-418f-a433-f851ab5a981c\") "
Nov 03 12:45:21.381617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.381600    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j9w4\" (UniqueName: \"kubernetes.io/projected/889e3dec-ac77-4225-af3d-cd780a6e5012-kube-api-access-5j9w4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:21.381713 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.381690    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca" pod "3ce4bff6-b99d-418f-a433-f851ab5a981c" (UID: "3ce4bff6-b99d-418f-a433-f851ab5a981c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:21.384062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.384033    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b" pod "3ce4bff6-b99d-418f-a433-f851ab5a981c" (UID: "3ce4bff6-b99d-418f-a433-f851ab5a981c"). InnerVolumeSpecName "kube-api-access-85d6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:21.486774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.482743    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85d6b\" (UniqueName: \"kubernetes.io/projected/3ce4bff6-b99d-418f-a433-f851ab5a981c-kube-api-access-85d6b\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:21.486774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.482803    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4bff6-b99d-418f-a433-f851ab5a981c-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:21.773269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.773185    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"
Nov 03 12:45:21.773269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.773184    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p" event={"ID":"889e3dec-ac77-4225-af3d-cd780a6e5012","Type":"ContainerDied","Data":"dbcb31576a75f0d0cd11ff4a20f7aef5a1230818617a818da19f5f94194faa0f"}
Nov 03 12:45:21.773269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.773264    4546 scope.go:122] "RemoveContainer" containerID="ff3822328c4819f8fd92420224c7303e3426ec8d89311fab1debd37718ba58be"
Nov 03 12:45:21.775843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.775811    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"
Nov 03 12:45:21.775843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.775829    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp" event={"ID":"3ce4bff6-b99d-418f-a433-f851ab5a981c","Type":"ContainerDied","Data":"c27cc2d6e1c73c0d5ee07c449c6cb3828893b3fc0b75a1d03fd1f36978d9359d"}
Nov 03 12:45:21.784348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.784331    4546 scope.go:122] "RemoveContainer" containerID="8329a9aeb593d43e629812e9b1195aee42c768b9bb75b315ee2e3c3c1b0fe38f"
Nov 03 12:45:21.927289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.927254    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"]
Nov 03 12:45:21.930505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:21.930476    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895165-22m5p"]
Nov 03 12:45:22.033136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:22.033050    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"]
Nov 03 12:45:22.037930 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:22.037905    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895165-6xszp"]
Nov 03 12:45:22.574198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:22.574169    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" path="/var/lib/kubelet/pods/3ce4bff6-b99d-418f-a433-f851ab5a981c/volumes"
Nov 03 12:45:22.574590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:22.574576    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" path="/var/lib/kubelet/pods/6a6f7a41-8107-4616-a8ba-db85a462c0ea/volumes"
Nov 03 12:45:22.574983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:22.574970    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" path="/var/lib/kubelet/pods/889e3dec-ac77-4225-af3d-cd780a6e5012/volumes"
Nov 03 12:45:22.575621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:22.575607    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" path="/var/lib/kubelet/pods/c5698d36-32e3-43a2-bcba-c30bbadf53c2/volumes"
Nov 03 12:45:37.653035 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:45:37.652935    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:45:56.043934 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:56.043903    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"]
Nov 03 12:45:56.092278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:56.092233    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d948386-lz45r"]
Nov 03 12:45:56.582167 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:45:56.582128    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59436701-d4cc-4fc5-9d0b-c9aeb383dffc" path="/var/lib/kubelet/pods/59436701-d4cc-4fc5-9d0b-c9aeb383dffc/volumes"
Nov 03 12:46:06.262806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.262772    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"]
Nov 03 12:46:06.263435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263388    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.263435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263410    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.263435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263424    4546 cpu_manager.go:414] "containerMap: removing container" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.263435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263433    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.263435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263447    4546 cpu_manager.go:414] "containerMap: removing container" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263455    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263473    4546 cpu_manager.go:414] "containerMap: removing container" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263479    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263488    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263495    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263510    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263516    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263527    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263534    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263543    4546 cpu_manager.go:414] "containerMap: removing container" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263550    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263561    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263567    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263577    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263583    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263595    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263602    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263612    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263619    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263632    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263639    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263649    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263656    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263666    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263673    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263696    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263702    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263716    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263723    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263735    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263743    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263773    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263780    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263790    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263797    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263815    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263822    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263838    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263845    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263875    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.263861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263883    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263892    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263899    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263909    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.263917    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264084    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264098    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264107    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264116    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264126    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264136    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264145    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264153    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264162    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264171    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264178    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264186    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264196    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264202    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264212    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264222    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264230    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264239    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264250    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264259    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce4bff6-b99d-418f-a433-f851ab5a981c" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264270    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.266042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.264280    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.269689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.269670    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.348611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.348575    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"]
Nov 03 12:46:06.371775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.369067    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-94e247\""
Nov 03 12:46:06.371775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.369269    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-94e247-dockercfg-qmqpb\""
Nov 03 12:46:06.428524 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.428492    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"]
Nov 03 12:46:06.429142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429121    4546 cpu_manager.go:414] "containerMap: removing container" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.429142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429140    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538e81a7-c04d-4fb3-b1c7-bf0377713c00" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429156    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429162    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429171    4546 cpu_manager.go:414] "containerMap: removing container" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429175    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429186    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429190    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429202    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429207    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429312    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce118ef-6431-467c-8b57-c7abfa319fea" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429322    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e58475d-bb7f-4806-8009-6a818555213b" containerName="cleanup"
Nov 03 12:46:06.429327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429331    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="889e3dec-ac77-4225-af3d-cd780a6e5012" containerName="cleanup"
Nov 03 12:46:06.429681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429338    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab114a94-6cfb-4da4-95fe-d196b40e3116" containerName="cleanup"
Nov 03 12:46:06.429681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429344    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6f7a41-8107-4616-a8ba-db85a462c0ea" containerName="cleanup"
Nov 03 12:46:06.429681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429350    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1720a598-042b-4683-b366-e760359b5826" containerName="cleanup"
Nov 03 12:46:06.429681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429359    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac85d3a5-4860-47bf-8d59-840673a22eb9" containerName="cleanup"
Nov 03 12:46:06.429681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.429365    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5698d36-32e3-43a2-bcba-c30bbadf53c2" containerName="cleanup"
Nov 03 12:46:06.433087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.433072    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.474017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.473987    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcdg\" (UniqueName: \"kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg\") pod \"payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.474172 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.474142    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.486279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.486245    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"]
Nov 03 12:46:06.576065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.575994    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.576065 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.576052    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.576298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.576137    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6gs\" (UniqueName: \"kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs\") pod \"payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.576298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.576224    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcdg\" (UniqueName: \"kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg\") pod \"payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.576868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.576844    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.629030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.629001    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcdg\" (UniqueName: \"kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg\") pod \"payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:06.676895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.676859    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.677058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.676964    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6gs\" (UniqueName: \"kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs\") pod \"payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.677492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.677471    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.732489 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.732456    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6gs\" (UniqueName: \"kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs\") pod \"payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.746576 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.746549    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:06.883399 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:06.883314    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:07.119921 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:07.119883    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"]
Nov 03 12:46:07.123971 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:46:07.123940    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e6c6a9_aef7_4848_8e9e_da053fa51568.slice/crio-91ebb335b4e1a35e1f44278f8e1d426810210454be021114d7f6d5056f85cddd WatchSource:0}: Error finding container 91ebb335b4e1a35e1f44278f8e1d426810210454be021114d7f6d5056f85cddd: Status 404 returned error can't find the container with id 91ebb335b4e1a35e1f44278f8e1d426810210454be021114d7f6d5056f85cddd
Nov 03 12:46:07.251133 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:07.251106    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"]
Nov 03 12:46:07.252348 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:46:07.252318    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057a30fe_c087_455a_9c93_68e38ed71dfb.slice/crio-b3e10b4bd743cff4ed494a824603c7a579a23b6e9517e59458c54dfd5c28dd48 WatchSource:0}: Error finding container b3e10b4bd743cff4ed494a824603c7a579a23b6e9517e59458c54dfd5c28dd48: Status 404 returned error can't find the container with id b3e10b4bd743cff4ed494a824603c7a579a23b6e9517e59458c54dfd5c28dd48
Nov 03 12:46:07.389112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:07.389080    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"]
Nov 03 12:46:07.389544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:07.389337    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy" containerID="cri-o://62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f" gracePeriod=30
Nov 03 12:46:07.652720 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:46:07.652685    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:46:08.114899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.114860    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" event={"ID":"72e6c6a9-aef7-4848-8e9e-da053fa51568","Type":"ContainerStarted","Data":"f57550c763ff0aa47a8d6021340a3a3929be096d2c38af436107f063121873ad"}
Nov 03 12:46:08.115139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.114928    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" event={"ID":"72e6c6a9-aef7-4848-8e9e-da053fa51568","Type":"ContainerStarted","Data":"91ebb335b4e1a35e1f44278f8e1d426810210454be021114d7f6d5056f85cddd"}
Nov 03 12:46:08.115139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.115044    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:08.116840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.116803    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" event={"ID":"057a30fe-c087-455a-9c93-68e38ed71dfb","Type":"ContainerStarted","Data":"bc18bebbea1b12a027545f201ddeede78f53c287a17caf99c5dc60552a2b8927"}
Nov 03 12:46:08.116975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.116844    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" event={"ID":"057a30fe-c087-455a-9c93-68e38ed71dfb","Type":"ContainerStarted","Data":"b3e10b4bd743cff4ed494a824603c7a579a23b6e9517e59458c54dfd5c28dd48"}
Nov 03 12:46:08.116975 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.116889    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:08.225121 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.225060    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" podStartSLOduration=2.225043739 podStartE2EDuration="2.225043739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:46:08.222065594 +0000 UTC m=+3309.793649521" watchObservedRunningTime="2026-11-03 12:46:08.225043739 +0000 UTC m=+3309.796627697"
Nov 03 12:46:08.330232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.330182    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" podStartSLOduration=2.330168846 podStartE2EDuration="2.330168846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:46:08.326704291 +0000 UTC m=+3309.898288203" watchObservedRunningTime="2026-11-03 12:46:08.330168846 +0000 UTC m=+3309.901752757"
Nov 03 12:46:08.574446 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.574413    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"]
Nov 03 12:46:08.578544 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.578525    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.648153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.647698    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ff65b5-openshift-default-dockercfg-7wp7d\""
Nov 03 12:46:08.716832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.664262    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"]
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.751585    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.751670    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.752638    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.752696    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.752740    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.752817    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.753054    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.753153    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw55k\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.764241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.753186    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855052 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.854942    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855052 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855022    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw55k\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855050    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855144    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855194    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855230    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855253    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855298    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855348    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855616 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855417    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.855971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855943    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.856073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.855951    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.856147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.856063    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.856800 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.856733    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.858182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.858150    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.858514 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.858492    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.908581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.908554    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:08.908581 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:08.908574    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw55k\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k\") pod \"e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") " pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:09.192303 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:09.192215    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:09.606397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:09.606356    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"]
Nov 03 12:46:09.606858 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:46:09.606643    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d68b570_3535_448b_954d_6c05e1300238.slice/crio-1d2517f4b3b1e346e15b912eed7af8318988d64710606810ca066e45c097cd17 WatchSource:0}: Error finding container 1d2517f4b3b1e346e15b912eed7af8318988d64710606810ca066e45c097cd17: Status 404 returned error can't find the container with id 1d2517f4b3b1e346e15b912eed7af8318988d64710606810ca066e45c097cd17
Nov 03 12:46:09.609533 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:09.609503    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:46:09.609625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:09.609564    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:46:09.609625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:09.609592    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:46:10.137921 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:10.137884    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" event={"ID":"6d68b570-3535-448b-954d-6c05e1300238","Type":"ContainerStarted","Data":"d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2"}
Nov 03 12:46:10.138112 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:10.137942    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" event={"ID":"6d68b570-3535-448b-954d-6c05e1300238","Type":"ContainerStarted","Data":"1d2517f4b3b1e346e15b912eed7af8318988d64710606810ca066e45c097cd17"}
Nov 03 12:46:10.193230 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:10.193207    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:10.199007 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:10.198988    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:10.248506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:10.248459    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" podStartSLOduration=2.248444543 podStartE2EDuration="2.248444543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:46:10.245110518 +0000 UTC m=+3311.816694448" watchObservedRunningTime="2026-11-03 12:46:10.248444543 +0000 UTC m=+3311.820028455"
Nov 03 12:46:11.144989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:11.144957    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:11.146025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:11.146005    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:12.660005 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.659947    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:46:12.701659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.701629    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.701821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.701687    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.701821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.701772    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.701821 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.701805    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.701989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.701844    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.702048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.701990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.702108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702046    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.702108 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702087    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.702235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702100    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.702235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702126    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:46:12.702235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702132    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr\") pod \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\" (UID: \"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d\") "
Nov 03 12:46:12.702235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702177    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.702471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702283    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.702471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702319    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.702744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702708    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.702744 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702732    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.702900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702764    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.702900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702780    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.702900 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.702795    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.704221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.704192    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:46:12.704221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.704195    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:12.704384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.704272    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "kube-api-access-5hqqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:46:12.704632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.704616    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token" pod "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" (UID: "ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:46:12.804124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.804085    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.804124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.804125    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqqr\" (UniqueName: \"kubernetes.io/projected/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-kube-api-access-5hqqr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.804338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.804141    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:12.804338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:12.804155    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:13.158998 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.158961    4546 generic.go:356] "Generic (PLEG): container finished" podID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerID="62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f" exitCode=0
Nov 03 12:46:13.159232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.159030    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"
Nov 03 12:46:13.159232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.159096    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" event={"ID":"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d","Type":"ContainerDied","Data":"62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f"}
Nov 03 12:46:13.159232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.159177    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc" event={"ID":"ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d","Type":"ContainerDied","Data":"57af53232add2fcc344fa66e3030d90496cc44c819549e51316c11f2d4286b69"}
Nov 03 12:46:13.159232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.159211    4546 scope.go:122] "RemoveContainer" containerID="62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f"
Nov 03 12:46:13.172722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.172702    4546 scope.go:122] "RemoveContainer" containerID="62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f"
Nov 03 12:46:13.173029 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:46:13.173010    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f\": container with ID starting with 62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f not found: ID does not exist" containerID="62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f"
Nov 03 12:46:13.173104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.173037    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f"} err="failed to get container status \"62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f\": rpc error: code = NotFound desc = could not find container \"62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f\": container with ID starting with 62779966ea7beefa6933ee707518b6df0c52e9ff1ecd5c835685a5f72cbc099f not found: ID does not exist"
Nov 03 12:46:13.314442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.314412    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"]
Nov 03 12:46:13.360426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:13.360397    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-52ab15-openshift-default-85cf87cc65-wj4mc"]
Nov 03 12:46:14.574058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:14.574027    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" path="/var/lib/kubelet/pods/ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d/volumes"
Nov 03 12:46:19.131251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:19.131221    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 12:46:19.131812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:19.131298    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 12:46:36.312220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:36.312184    4546 generic.go:356] "Generic (PLEG): container finished" podID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerID="9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9" exitCode=0
Nov 03 12:46:36.312803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:36.312261    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" event={"ID":"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342","Type":"ContainerDied","Data":"9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"}
Nov 03 12:46:36.312895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:36.312802    4546 scope.go:122] "RemoveContainer" containerID="9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"
Nov 03 12:46:36.314549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:36.314522    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:46:36.314621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:36.314591    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:46:36.314662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:36.314624    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:46:37.338659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:37.338619    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" event={"ID":"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342","Type":"ContainerStarted","Data":"c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7"}
Nov 03 12:46:37.491909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:37.491856    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:46:37.492139 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:37.492114    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Nov 03 12:46:37.492234 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:37.492171    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Nov 03 12:46:37.658940 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:46:37.658853    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:46:38.492622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:38.492590    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Nov 03 12:46:38.493049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:38.492638    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Nov 03 12:46:39.492818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:39.492782    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Nov 03 12:46:39.493250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:39.492850    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Nov 03 12:46:40.492412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:40.492374    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Nov 03 12:46:40.492598 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:40.492431    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Nov 03 12:46:41.492728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:41.492690    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Nov 03 12:46:41.493164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:41.492772    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Nov 03 12:46:42.510015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:42.509985    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:46:42.510537 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:42.510382    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:46:42.511210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:42.511188    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:46:53.381812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:53.381777    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"]
Nov 03 12:46:53.382266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:53.381997    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" podUID="6d68b570-3535-448b-954d-6c05e1300238" containerName="istio-proxy" containerID="cri-o://d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2" gracePeriod=30
Nov 03 12:46:56.148103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:56.148063    4546 patch_prober.go:29] interesting pod/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:46:56.148496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:56.148138    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" podUID="6d68b570-3535-448b-954d-6c05e1300238" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:46:58.690595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.690560    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:58.744726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.744695    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.744997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.744796    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.744997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.744887    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.744997 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.744960    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.745171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.744999    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.745171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745063    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.745171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745104    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.745171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745148    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.745387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745188    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket\") pod \"6d68b570-3535-448b-954d-6c05e1300238\" (UID: \"6d68b570-3535-448b-954d-6c05e1300238\") "
Nov 03 12:46:58.745387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745058    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:58.745387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745251    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:58.745548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745413    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:58.745548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745419    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:46:58.745660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745570    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:58.746093 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.745995    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.746193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.746145    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68b570-3535-448b-954d-6c05e1300238-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.746193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.746160    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.746193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.746174    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.746193 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.746188    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.747661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.747634    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:46:58.747809 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.747666    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:46:58.747977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.747944    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "kube-api-access-gw55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:46:58.747977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.747954    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy" pod "6d68b570-3535-448b-954d-6c05e1300238" (UID: "6d68b570-3535-448b-954d-6c05e1300238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:46:58.847266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.847236    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68b570-3535-448b-954d-6c05e1300238-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.847266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.847263    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.847266 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.847273    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68b570-3535-448b-954d-6c05e1300238-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:58.847521 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:58.847285    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw55k\" (UniqueName: \"kubernetes.io/projected/6d68b570-3535-448b-954d-6c05e1300238-kube-api-access-gw55k\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:46:59.483062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.483024    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d68b570-3535-448b-954d-6c05e1300238" containerID="d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2" exitCode=0
Nov 03 12:46:59.483300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.483076    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"
Nov 03 12:46:59.483300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.483136    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" event={"ID":"6d68b570-3535-448b-954d-6c05e1300238","Type":"ContainerDied","Data":"d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2"}
Nov 03 12:46:59.483300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.483203    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm" event={"ID":"6d68b570-3535-448b-954d-6c05e1300238","Type":"ContainerDied","Data":"1d2517f4b3b1e346e15b912eed7af8318988d64710606810ca066e45c097cd17"}
Nov 03 12:46:59.483300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.483237    4546 scope.go:122] "RemoveContainer" containerID="d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2"
Nov 03 12:46:59.494471 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.494452    4546 scope.go:122] "RemoveContainer" containerID="d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2"
Nov 03 12:46:59.494824 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:46:59.494802    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2\": container with ID starting with d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2 not found: ID does not exist" containerID="d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2"
Nov 03 12:46:59.494898 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.494836    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2"} err="failed to get container status \"d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2\": rpc error: code = NotFound desc = could not find container \"d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2\": container with ID starting with d3e6eb06e46cfdb56c1ef8a25588b8c64a2225bbbd58514a5bdbc11930b1ffd2 not found: ID does not exist"
Nov 03 12:46:59.683623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.683583    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"]
Nov 03 12:46:59.729803 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:46:59.729765    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ff65b5-openshift-default-7d44bf86d5-4pbrm"]
Nov 03 12:47:00.580287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:00.580248    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d68b570-3535-448b-954d-6c05e1300238" path="/var/lib/kubelet/pods/6d68b570-3535-448b-954d-6c05e1300238/volumes"
Nov 03 12:47:01.411667 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:01.411626    4546 scope.go:122] "RemoveContainer" containerID="e0a4169953f5e42e4cc76b746194d2957a24b501d942068e001c364244c47232"
Nov 03 12:47:02.962078 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962030    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"]
Nov 03 12:47:02.962623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962608    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy"
Nov 03 12:47:02.962679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962625    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy"
Nov 03 12:47:02.962679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962648    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d68b570-3535-448b-954d-6c05e1300238" containerName="istio-proxy"
Nov 03 12:47:02.962679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962653    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d68b570-3535-448b-954d-6c05e1300238" containerName="istio-proxy"
Nov 03 12:47:02.962814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962776    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbb5ce2-9e39-42c5-b0c4-ee64be463a2d" containerName="istio-proxy"
Nov 03 12:47:02.962814 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.962788    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d68b570-3535-448b-954d-6c05e1300238" containerName="istio-proxy"
Nov 03 12:47:02.968038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:02.968018    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.018330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.018297    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-6fb3a8-openshift-default-dockercfg-j9znp\""
Nov 03 12:47:03.068235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.068201    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"]
Nov 03 12:47:03.119794 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.119732    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.119999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.119818    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcgz\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.119999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.119865    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.120116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.119991    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.120116 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.120033    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.120220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.120138    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.120342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.120268    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.120342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.120306    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.120342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.120330    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222102 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222064    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222121    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222157    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222204    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222256    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcgz\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222311 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222303    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222375    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222414    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222479    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222678    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.222942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222919    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.223033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.222977    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.223084 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.223064    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.223127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.223086    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.224679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.224650    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.225424 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.225404    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.276509 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.276476    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcgz\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.276672 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.276611    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.280574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.280552    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:03.705090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.705052    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"]
Nov 03 12:47:03.706427 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:47:03.706402    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f552f2f_d534_499d_827c_093c8c4f1b22.slice/crio-0bd50fd62337bc9e9346604ca91be0809b75337e1d83885cc033ceac04480861 WatchSource:0}: Error finding container 0bd50fd62337bc9e9346604ca91be0809b75337e1d83885cc033ceac04480861: Status 404 returned error can't find the container with id 0bd50fd62337bc9e9346604ca91be0809b75337e1d83885cc033ceac04480861
Nov 03 12:47:03.709518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.709480    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:47:03.709606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.709557    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:47:03.709606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:03.709589    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:47:04.538906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:04.538870    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" event={"ID":"9f552f2f-d534-499d-827c-093c8c4f1b22","Type":"ContainerStarted","Data":"ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5"}
Nov 03 12:47:04.539404 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:04.538933    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" event={"ID":"9f552f2f-d534-499d-827c-093c8c4f1b22","Type":"ContainerStarted","Data":"0bd50fd62337bc9e9346604ca91be0809b75337e1d83885cc033ceac04480861"}
Nov 03 12:47:04.740215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:04.740154    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podStartSLOduration=2.740139084 podStartE2EDuration="2.740139084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:04.693128871 +0000 UTC m=+3366.264712806" watchObservedRunningTime="2026-11-03 12:47:04.740139084 +0000 UTC m=+3366.311722994"
Nov 03 12:47:05.280942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:05.280902    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:05.282639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:05.282610    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 12:47:05.282822 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:05.282662    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 12:47:06.281543 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:06.281505    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 12:47:06.282017 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:06.281565    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 12:47:07.281631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:07.281598    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 12:47:07.282054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:07.281649    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 12:47:07.660729 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:47:07.660619    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:47:08.281716 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:08.281687    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 12:47:08.282176 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:08.281737    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 12:47:09.281577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:09.281540    4546 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 12:47:09.281884 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:09.281595    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 12:47:10.286263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:10.286235    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:10.286690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:10.286623    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:10.287410 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:10.287389    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 12:47:12.704774 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.703932    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"]
Nov 03 12:47:12.709849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.709716    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:12.760671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.760639    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"]
Nov 03 12:47:12.802617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.802586    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-aa20ad\""
Nov 03 12:47:12.802617 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.802587    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-aa20ad-dockercfg-h4j7k\""
Nov 03 12:47:12.873238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.873200    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:12.873427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.873250    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qjg\" (UniqueName: \"kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg\") pod \"payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:12.974541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.974509    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:12.974741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.974564    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qjg\" (UniqueName: \"kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg\") pod \"payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:12.975286 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:12.975263    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:13.027942 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.027904    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qjg\" (UniqueName: \"kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg\") pod \"payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:13.036029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.036000    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:13.407633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.407600    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"]
Nov 03 12:47:13.408030 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:47:13.408001    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7d3088_f7e9_414c_842c_8ff2d13c44e6.slice/crio-81363cbf13826f11ecfb13fe764de01c9185488a1bfd97407e8609fcfcfa9d20 WatchSource:0}: Error finding container 81363cbf13826f11ecfb13fe764de01c9185488a1bfd97407e8609fcfcfa9d20: Status 404 returned error can't find the container with id 81363cbf13826f11ecfb13fe764de01c9185488a1bfd97407e8609fcfcfa9d20
Nov 03 12:47:13.628247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.628207    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4" event={"ID":"bb7d3088-f7e9-414c-842c-8ff2d13c44e6","Type":"ContainerStarted","Data":"b702e16b2e11954d235c80efbe06582112575f5c69e87b96b33bc404bb044d4e"}
Nov 03 12:47:13.628437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.628276    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:13.628437 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.628294    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4" event={"ID":"bb7d3088-f7e9-414c-842c-8ff2d13c44e6","Type":"ContainerStarted","Data":"81363cbf13826f11ecfb13fe764de01c9185488a1bfd97407e8609fcfcfa9d20"}
Nov 03 12:47:13.735356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:13.735305    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4" podStartSLOduration=1.735290419 podStartE2EDuration="1.735290419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:13.733631732 +0000 UTC m=+3375.305215644" watchObservedRunningTime="2026-11-03 12:47:13.735290419 +0000 UTC m=+3375.306874377"
Nov 03 12:47:24.638220 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:24.638192    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:47:26.085082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:26.085043    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"]
Nov 03 12:47:26.130341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:26.130307    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2673e26-q59h8"]
Nov 03 12:47:26.574251 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:26.574220    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7c8ffd-8c04-41f3-9b15-b2b52b0253de" path="/var/lib/kubelet/pods/6d7c8ffd-8c04-41f3-9b15-b2b52b0253de/volumes"
Nov 03 12:47:30.731432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.731400    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"]
Nov 03 12:47:30.735741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.735718    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:30.806866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.806409    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfsdg\" (UniqueName: \"kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg\") pod \"payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:30.806866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.806588    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:30.834505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.834470    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c84ae1-dockercfg-qjljq\""
Nov 03 12:47:30.834728 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.834541    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c84ae1\""
Nov 03 12:47:30.837061 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.837031    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"]
Nov 03 12:47:30.903730 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.903695    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"]
Nov 03 12:47:30.907522 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.907499    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:30.907661 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.907616    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfsdg\" (UniqueName: \"kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg\") pod \"payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:30.907941 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.907926    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:30.908460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.908439    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:30.954982 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.954954    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"]
Nov 03 12:47:30.969507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:30.969461    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfsdg\" (UniqueName: \"kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg\") pod \"payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:31.009128 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.009045    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6ms\" (UniqueName: \"kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms\") pod \"payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.009278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.009252    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.050239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.050216    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:31.111009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.110977    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.111154 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.111094    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6ms\" (UniqueName: \"kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms\") pod \"payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.112223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.112197    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.164235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.164203    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6ms\" (UniqueName: \"kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms\") pod \"payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.221940 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.221910    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.422039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.422007    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"]
Nov 03 12:47:31.423133 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:47:31.423105    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode750a5cb_cfda_4b70_a4bf_ed374dc9a2ba.slice/crio-d9a8d7ee93299aef2fd05701957aafe6e1978f10707ba08067e3b50a3e06a4b2 WatchSource:0}: Error finding container d9a8d7ee93299aef2fd05701957aafe6e1978f10707ba08067e3b50a3e06a4b2: Status 404 returned error can't find the container with id d9a8d7ee93299aef2fd05701957aafe6e1978f10707ba08067e3b50a3e06a4b2
Nov 03 12:47:31.588282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.588258    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"]
Nov 03 12:47:31.589168 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:47:31.589138    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8deefe_bc86_4a39_ad4b_867441fb9a19.slice/crio-8bb94413316d0de2fd61150c276e3ea1fd7928202a2cf32e7ac2bdbdf170fa66 WatchSource:0}: Error finding container 8bb94413316d0de2fd61150c276e3ea1fd7928202a2cf32e7ac2bdbdf170fa66: Status 404 returned error can't find the container with id 8bb94413316d0de2fd61150c276e3ea1fd7928202a2cf32e7ac2bdbdf170fa66
Nov 03 12:47:31.747985 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.747944    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62" event={"ID":"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba","Type":"ContainerStarted","Data":"2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060"}
Nov 03 12:47:31.748494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.748008    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62" event={"ID":"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba","Type":"ContainerStarted","Data":"d9a8d7ee93299aef2fd05701957aafe6e1978f10707ba08067e3b50a3e06a4b2"}
Nov 03 12:47:31.748494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.748197    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:31.749801 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.749777    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899" event={"ID":"5b8deefe-bc86-4a39-ad4b-867441fb9a19","Type":"ContainerStarted","Data":"8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf"}
Nov 03 12:47:31.749926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.749823    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:47:31.749926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.749848    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899" event={"ID":"5b8deefe-bc86-4a39-ad4b-867441fb9a19","Type":"ContainerStarted","Data":"8bb94413316d0de2fd61150c276e3ea1fd7928202a2cf32e7ac2bdbdf170fa66"}
Nov 03 12:47:31.856559 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.856460    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62" podStartSLOduration=1.856443628 podStartE2EDuration="1.856443628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:31.852879749 +0000 UTC m=+3393.424463662" watchObservedRunningTime="2026-11-03 12:47:31.856443628 +0000 UTC m=+3393.428027541"
Nov 03 12:47:31.960644 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:31.960589    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899" podStartSLOduration=1.960574794 podStartE2EDuration="1.960574794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:31.957956657 +0000 UTC m=+3393.529540572" watchObservedRunningTime="2026-11-03 12:47:31.960574794 +0000 UTC m=+3393.532158706"
Nov 03 12:47:37.043434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:37.043398    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"]
Nov 03 12:47:37.089707 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:37.089665    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ee0566-dstl8"]
Nov 03 12:47:37.651814 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:47:37.651771    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:47:38.104449 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.104416    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"]
Nov 03 12:47:38.108978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.108956    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.202789    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.202854    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203001    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203045    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203175    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203260    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203325    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203373    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.205487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.203415    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdfb\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.207031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.207004    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-38f178-openshift-default-dockercfg-cjwq6\""
Nov 03 12:47:38.209297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.209267    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"]
Nov 03 12:47:38.306109 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306067    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306133    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306170    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306210    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306327 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306288    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306347    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306398    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdfb\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306512    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.306569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.306547    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.325908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.323590    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.325908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.323587    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.325908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.323950    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.325908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.324978    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.326315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.325927    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.331282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.328548    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.337811 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.332303    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.381036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.380917    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.382160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.382131    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdfb\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb\") pod \"e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") " pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.424624 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.424591    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:38.574986 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.574955    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07677acf-8e3c-4756-9a03-243fdc1b9fca" path="/var/lib/kubelet/pods/07677acf-8e3c-4756-9a03-243fdc1b9fca/volumes"
Nov 03 12:47:38.844059 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.844027    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"]
Nov 03 12:47:38.845342 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:47:38.845317    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3141d97b_c167_49c0_b1e6_2596452f6117.slice/crio-c6ff2b3334cc250e2f0097bbe9d4ab22490bb1aa1f480de49e6ac4808e88d16e WatchSource:0}: Error finding container c6ff2b3334cc250e2f0097bbe9d4ab22490bb1aa1f480de49e6ac4808e88d16e: Status 404 returned error can't find the container with id c6ff2b3334cc250e2f0097bbe9d4ab22490bb1aa1f480de49e6ac4808e88d16e
Nov 03 12:47:38.848079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.848045    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:47:38.848191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.848127    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:47:38.848191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:38.848172    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:47:39.804165 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:39.804125    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" event={"ID":"3141d97b-c167-49c0-b1e6-2596452f6117","Type":"ContainerStarted","Data":"e37ab13ea39c3e41aedbaf51559efa7a54038307a722eebc231ebd2f143e7ff0"}
Nov 03 12:47:39.804603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:39.804227    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" event={"ID":"3141d97b-c167-49c0-b1e6-2596452f6117","Type":"ContainerStarted","Data":"c6ff2b3334cc250e2f0097bbe9d4ab22490bb1aa1f480de49e6ac4808e88d16e"}
Nov 03 12:47:40.425846 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:40.425812    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:40.431257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:40.431233    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:40.541215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:40.541165    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podStartSLOduration=2.541149402 podStartE2EDuration="2.541149402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:40.004247067 +0000 UTC m=+3401.575830980" watchObservedRunningTime="2026-11-03 12:47:40.541149402 +0000 UTC m=+3402.112733321"
Nov 03 12:47:40.815908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:40.815881    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:40.816936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:40.816912    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:47:42.759545 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:42.759515    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:47:42.760035 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:47:42.759932    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:48:03.043732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:03.043700    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"]
Nov 03 12:48:03.089698 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:03.089668    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e431771c-ngtc5"]
Nov 03 12:48:04.574002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:04.573969    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7191e0c-201a-4188-8509-7bffe2d42858" path="/var/lib/kubelet/pods/f7191e0c-201a-4188-8509-7bffe2d42858/volumes"
Nov 03 12:48:07.652658 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:48:07.652624    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:48:14.043494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:14.043457    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"]
Nov 03 12:48:14.089090 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:14.089058    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae09438-fvw6l"]
Nov 03 12:48:14.573894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:14.573864    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fcfb08-31ca-4f36-a229-be7d9cc5aa8a" path="/var/lib/kubelet/pods/46fcfb08-31ca-4f36-a229-be7d9cc5aa8a/volumes"
Nov 03 12:48:36.045288 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:36.045248    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"]
Nov 03 12:48:36.091068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:36.091043    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18a55b2-9rgl4"]
Nov 03 12:48:36.574337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:36.574304    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4c187a-7c5a-414e-8694-2d59a5005774" path="/var/lib/kubelet/pods/5c4c187a-7c5a-414e-8694-2d59a5005774/volumes"
Nov 03 12:48:37.651843 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:48:37.651765    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:48:47.045366 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:47.045328    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"]
Nov 03 12:48:47.089168 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:47.089134    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62c3613-k6rbq"]
Nov 03 12:48:48.574321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:48.574284    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9e2d3f-7447-45a1-a9d7-2495e535cd13" path="/var/lib/kubelet/pods/0f9e2d3f-7447-45a1-a9d7-2495e535cd13/volumes"
Nov 03 12:48:57.467308 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.467271    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"]
Nov 03 12:48:57.473138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.473117    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.562787 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.562737    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.562976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.562865    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.562976 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.562908    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p87\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.563072 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.563058    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.569653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.569623    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"kube-root-ca.crt\""
Nov 03 12:48:57.569653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.569648    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"payload-processing-external-model-plugins-e2e-extmod-b-94e247\""
Nov 03 12:48:57.569929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.569667    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"agc-cred-8bea6cf1ca2b\""
Nov 03 12:48:57.569929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.569690    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"openshift-service-ca.crt\""
Nov 03 12:48:57.570484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.570467    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"routing-overlay\""
Nov 03 12:48:57.570566 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.570508    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"payload-processing-external-model-e2e--332b9201-dockercfg-9cmzl\""
Nov 03 12:48:57.574104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.574082    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"]
Nov 03 12:48:57.664343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.664305    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.664556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.664442    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.664556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.664508    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.664556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.664537    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p87\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.665127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.665092    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.665349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.665328    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.667064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.667045    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.718228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.718151    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p87\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:57.787008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:57.786977    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:58.208865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:58.208834    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"]
Nov 03 12:48:58.210211 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:48:58.210184    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1914aac_6091_40b0_924d_a5fb9b13321c.slice/crio-3304588b7b9fc7bd88e8c4ef42abc7d0d9b22f44e8b06f7b7e461ccf94d8f519 WatchSource:0}: Error finding container 3304588b7b9fc7bd88e8c4ef42abc7d0d9b22f44e8b06f7b7e461ccf94d8f519: Status 404 returned error can't find the container with id 3304588b7b9fc7bd88e8c4ef42abc7d0d9b22f44e8b06f7b7e461ccf94d8f519
Nov 03 12:48:58.340825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:58.340784    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn" event={"ID":"d1914aac-6091-40b0-924d-a5fb9b13321c","Type":"ContainerStarted","Data":"dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1"}
Nov 03 12:48:58.341015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:58.340856    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn" event={"ID":"d1914aac-6091-40b0-924d-a5fb9b13321c","Type":"ContainerStarted","Data":"3304588b7b9fc7bd88e8c4ef42abc7d0d9b22f44e8b06f7b7e461ccf94d8f519"}
Nov 03 12:48:58.341015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:58.340882    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:48:58.450216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:48:58.450156    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn" podStartSLOduration=1.450138262 podStartE2EDuration="1.450138262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:48:58.446194819 +0000 UTC m=+3480.017778729" watchObservedRunningTime="2026-11-03 12:48:58.450138262 +0000 UTC m=+3480.021722174"
Nov 03 12:49:01.458315 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:49:01.458281    4546 scope.go:122] "RemoveContainer" containerID="81670e61f9000d75cc31a0caea5c0c5512841d9bf6dd6b3699f4629df4403a06"
Nov 03 12:49:01.468136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:49:01.468113    4546 scope.go:122] "RemoveContainer" containerID="b2231ce2899a6bb853dfd90c66b4022203f3ff499e18825534bd1146c8c8cacb"
Nov 03 12:49:01.477305 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:49:01.477284    4546 scope.go:122] "RemoveContainer" containerID="15ad5bc7722c87f37a84e1de26e8d5a1a062d554610ae1739ecd3e8ef1a0106b"
Nov 03 12:49:07.664507 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:49:07.664464    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:49:09.351853 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:49:09.351816    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:49:37.652847 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:49:37.652811    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:50:01.511223 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.511193    4546 scope.go:122] "RemoveContainer" containerID="7dc72d06f72b1960832750d5615dd2cb9c2de3bbbe8bb7c39de53db0d251fbd7"
Nov 03 12:50:01.521590 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.521563    4546 scope.go:122] "RemoveContainer" containerID="4b128d16761124668963dc85c9214f2b92cdaab223a01378d3065383f3f2a8a9"
Nov 03 12:50:01.581907 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.581879    4546 scope.go:122] "RemoveContainer" containerID="9c1613b74c527d425351dc6bb61ad63ffea90af175bf83f06c517a3d4dbfe1cb"
Nov 03 12:50:01.803977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.803892    4546 generic.go:356] "Generic (PLEG): container finished" podID="3141d97b-c167-49c0-b1e6-2596452f6117" containerID="e37ab13ea39c3e41aedbaf51559efa7a54038307a722eebc231ebd2f143e7ff0" exitCode=0
Nov 03 12:50:01.803977 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.803917    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" event={"ID":"3141d97b-c167-49c0-b1e6-2596452f6117","Type":"ContainerDied","Data":"e37ab13ea39c3e41aedbaf51559efa7a54038307a722eebc231ebd2f143e7ff0"}
Nov 03 12:50:01.804473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.804457    4546 scope.go:122] "RemoveContainer" containerID="e37ab13ea39c3e41aedbaf51559efa7a54038307a722eebc231ebd2f143e7ff0"
Nov 03 12:50:01.806338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.806309    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:50:01.806425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.806380    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:50:01.806425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:01.806414    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:50:02.835795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:02.835736    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" event={"ID":"3141d97b-c167-49c0-b1e6-2596452f6117","Type":"ContainerStarted","Data":"b2fb29d4b204d5a11992145403917dd3a3e98f057a426d077f12cc71f2731aa0"}
Nov 03 12:50:03.428775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:03.428236    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:50:03.428775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:03.428365    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 12:50:03.428775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:03.428415    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 12:50:04.431777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:04.430181    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 12:50:04.431777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:04.430259    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 12:50:05.429382 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:05.429339    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 12:50:05.430497 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:05.430464    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 12:50:06.432222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:06.432186    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 12:50:06.432658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:06.432241    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 12:50:07.428778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:07.426184    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 12:50:07.428778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:07.426269    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 12:50:07.658309 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:50:07.658227    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:50:08.430270 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:08.430242    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:50:08.430671 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:08.430650    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:50:08.431469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:08.431453    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:50:14.653314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:14.653283    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"]
Nov 03 12:50:14.755012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:14.754974    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"]
Nov 03 12:50:19.654239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.654200    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl" podUID="2ee2bf08-7529-4e98-8de0-e985aa54ffcb" containerName="payload-processing" containerID="cri-o://cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176" gracePeriod=25
Nov 03 12:50:19.755325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.755292    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2" podUID="33462d86-9e7c-4129-8559-d0a32085014b" containerName="payload-pre-processing" containerID="cri-o://930d97070cbf71bf7e7fc6164ce8d89c5dda215c2cc1396251b9066802c8a1e1" gracePeriod=25
Nov 03 12:50:19.922300 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.922227    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:50:19.954331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.954299    4546 generic.go:356] "Generic (PLEG): container finished" podID="2ee2bf08-7529-4e98-8de0-e985aa54ffcb" containerID="cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176" exitCode=0
Nov 03 12:50:19.954500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.954350    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"
Nov 03 12:50:19.954500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.954386    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl" event={"ID":"2ee2bf08-7529-4e98-8de0-e985aa54ffcb","Type":"ContainerDied","Data":"cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176"}
Nov 03 12:50:19.954500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.954461    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl" event={"ID":"2ee2bf08-7529-4e98-8de0-e985aa54ffcb","Type":"ContainerDied","Data":"0b4b02f68a7d617604f302bb788d25d3b187de1c1413e04a557fbfc0c9239d99"}
Nov 03 12:50:19.954500 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.954479    4546 scope.go:122] "RemoveContainer" containerID="cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176"
Nov 03 12:50:19.956373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.956353    4546 generic.go:356] "Generic (PLEG): container finished" podID="33462d86-9e7c-4129-8559-d0a32085014b" containerID="930d97070cbf71bf7e7fc6164ce8d89c5dda215c2cc1396251b9066802c8a1e1" exitCode=0
Nov 03 12:50:19.956478 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.956419    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2" event={"ID":"33462d86-9e7c-4129-8559-d0a32085014b","Type":"ContainerDied","Data":"930d97070cbf71bf7e7fc6164ce8d89c5dda215c2cc1396251b9066802c8a1e1"}
Nov 03 12:50:19.965872 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.965851    4546 scope.go:122] "RemoveContainer" containerID="cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176"
Nov 03 12:50:19.966204 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:50:19.966178    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176\": container with ID starting with cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176 not found: ID does not exist" containerID="cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176"
Nov 03 12:50:19.966291 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:19.966216    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176"} err="failed to get container status \"cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176\": rpc error: code = NotFound desc = could not find container \"cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176\": container with ID starting with cdc5e8d4a3d4fc71ec51c9487f9fae5e520cd1509db9b29358463280cd8e1176 not found: ID does not exist"
Nov 03 12:50:20.030091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.030056    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume\") pod \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") "
Nov 03 12:50:20.030302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.030213    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls\" (UniqueName: \"kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls\") pod \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\" (UID: \"2ee2bf08-7529-4e98-8de0-e985aa54ffcb\") "
Nov 03 12:50:20.030479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.030454    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume" pod "2ee2bf08-7529-4e98-8de0-e985aa54ffcb" (UID: "2ee2bf08-7529-4e98-8de0-e985aa54ffcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:20.031385 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.030812    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:20.037283 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.037250    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls" pod "2ee2bf08-7529-4e98-8de0-e985aa54ffcb" (UID: "2ee2bf08-7529-4e98-8de0-e985aa54ffcb"). InnerVolumeSpecName "kube-api-access-bskls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:20.059345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.059315    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:50:20.131846 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.131822    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume\") pod \"33462d86-9e7c-4129-8559-d0a32085014b\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") "
Nov 03 12:50:20.132033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.131861    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc\" (UniqueName: \"kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc\") pod \"33462d86-9e7c-4129-8559-d0a32085014b\" (UID: \"33462d86-9e7c-4129-8559-d0a32085014b\") "
Nov 03 12:50:20.132192 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.132171    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume" pod "33462d86-9e7c-4129-8559-d0a32085014b" (UID: "33462d86-9e7c-4129-8559-d0a32085014b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:20.132431 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.132417    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskls\" (UniqueName: \"kubernetes.io/projected/2ee2bf08-7529-4e98-8de0-e985aa54ffcb-kube-api-access-bskls\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:20.132484 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.132434    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33462d86-9e7c-4129-8559-d0a32085014b-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:20.134129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.134109    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc" pod "33462d86-9e7c-4129-8559-d0a32085014b" (UID: "33462d86-9e7c-4129-8559-d0a32085014b"). InnerVolumeSpecName "kube-api-access-w4fpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:20.233880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.233850    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4fpc\" (UniqueName: \"kubernetes.io/projected/33462d86-9e7c-4129-8559-d0a32085014b-kube-api-access-w4fpc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:20.369525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.369489    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"]
Nov 03 12:50:20.416384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.416354    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bebc6d-67d4979767-xg6dl"]
Nov 03 12:50:20.573419 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.573341    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ee2bf08-7529-4e98-8de0-e985aa54ffcb" path="/var/lib/kubelet/pods/2ee2bf08-7529-4e98-8de0-e985aa54ffcb/volumes"
Nov 03 12:50:20.963653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.963566    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"
Nov 03 12:50:20.963653 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.963609    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2" event={"ID":"33462d86-9e7c-4129-8559-d0a32085014b","Type":"ContainerDied","Data":"8cf93db0f08bf7c95b569ef1c08667c081bb9ccc8250256e935e25c38b8a2933"}
Nov 03 12:50:20.964278 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:20.963683    4546 scope.go:122] "RemoveContainer" containerID="930d97070cbf71bf7e7fc6164ce8d89c5dda215c2cc1396251b9066802c8a1e1"
Nov 03 12:50:21.118279 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:21.118245    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"]
Nov 03 12:50:21.164433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:21.164404    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bebc6d-849bd5b447-6l6r2"]
Nov 03 12:50:22.573412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:22.573379    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33462d86-9e7c-4129-8559-d0a32085014b" path="/var/lib/kubelet/pods/33462d86-9e7c-4129-8559-d0a32085014b/volumes"
Nov 03 12:50:33.362342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:33.362294    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"]
Nov 03 12:50:37.651730 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:50:37.651687    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:50:38.362708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.362664    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv" podUID="6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" containerName="payload-pre-processing" containerID="cri-o://f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3" gracePeriod=25
Nov 03 12:50:38.637785 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.637737    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:50:38.662806 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.662775    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"]
Nov 03 12:50:38.663574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663551    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33462d86-9e7c-4129-8559-d0a32085014b" containerName="payload-pre-processing"
Nov 03 12:50:38.663679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663575    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33462d86-9e7c-4129-8559-d0a32085014b" containerName="payload-pre-processing"
Nov 03 12:50:38.663679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663625    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2ee2bf08-7529-4e98-8de0-e985aa54ffcb" containerName="payload-processing"
Nov 03 12:50:38.663679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663634    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee2bf08-7529-4e98-8de0-e985aa54ffcb" containerName="payload-processing"
Nov 03 12:50:38.663679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663659    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" containerName="payload-pre-processing"
Nov 03 12:50:38.663679 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663667    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" containerName="payload-pre-processing"
Nov 03 12:50:38.663969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663854    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" containerName="payload-pre-processing"
Nov 03 12:50:38.663969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663883    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33462d86-9e7c-4129-8559-d0a32085014b" containerName="payload-pre-processing"
Nov 03 12:50:38.663969 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.663897    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee2bf08-7529-4e98-8de0-e985aa54ffcb" containerName="payload-processing"
Nov 03 12:50:38.667365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.667343    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.716605 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.716578    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94e247\"/\"agc-cred-fdabd14ba96e\""
Nov 03 12:50:38.722463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.722428    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"]
Nov 03 12:50:38.735002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.734978    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume\") pod \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") "
Nov 03 12:50:38.735149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735009    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9\" (UniqueName: \"kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9\") pod \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\" (UID: \"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c\") "
Nov 03 12:50:38.735434 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735406    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume" pod "6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" (UID: "6df1a2fe-70e8-49de-abdc-ca9c1450dc7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:38.735562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735489    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.735629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735568    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.735692 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735669    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs987\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.735938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735811    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.735938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.735914    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:38.737387 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.737368    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9" pod "6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" (UID: "6df1a2fe-70e8-49de-abdc-ca9c1450dc7c"). InnerVolumeSpecName "kube-api-access-rbxh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:38.836663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.836630    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.836901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.836686    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.836901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.836717    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs987\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.836901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.836824    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.836901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.836895    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxh9\" (UniqueName: \"kubernetes.io/projected/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c-kube-api-access-rbxh9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:38.837384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.837358    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.837561 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.837514    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.839492 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.839470    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.909086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.909012    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs987\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987\") pod \"payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") " pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:38.981020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:38.980984    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:39.107296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.107262    4546 generic.go:356] "Generic (PLEG): container finished" podID="6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" containerID="f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3" exitCode=0
Nov 03 12:50:39.107487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.107304    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"
Nov 03 12:50:39.107487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.107342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv" event={"ID":"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c","Type":"ContainerDied","Data":"f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3"}
Nov 03 12:50:39.107487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.107407    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv" event={"ID":"6df1a2fe-70e8-49de-abdc-ca9c1450dc7c","Type":"ContainerDied","Data":"4269e3f829eeced0df491b2dcb7697082f0edd2ba80e14326d0e419619aeb05f"}
Nov 03 12:50:39.107487 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.107437    4546 scope.go:122] "RemoveContainer" containerID="f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3"
Nov 03 12:50:39.118915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.118897    4546 scope.go:122] "RemoveContainer" containerID="f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3"
Nov 03 12:50:39.119163 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:50:39.119144    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3\": container with ID starting with f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3 not found: ID does not exist" containerID="f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3"
Nov 03 12:50:39.119225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.119173    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3"} err="failed to get container status \"f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3\": rpc error: code = NotFound desc = could not find container \"f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3\": container with ID starting with f5f54d0fcc921487a6ae972a0baf34fb51a8c555be278bd3b8903516b21fc1f3 not found: ID does not exist"
Nov 03 12:50:39.260865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.260767    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"]
Nov 03 12:50:39.264182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.264153    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fd7881-c8f5c4dc-rnnqv"]
Nov 03 12:50:39.406181 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:50:39.406136    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddd777a_8170_415c_bc5e_47b844c113bb.slice/crio-5cc912cd8c45d8c807aa10c041e4361b9a287ffd1d903c4fb6a3cf2733758208 WatchSource:0}: Error finding container 5cc912cd8c45d8c807aa10c041e4361b9a287ffd1d903c4fb6a3cf2733758208: Status 404 returned error can't find the container with id 5cc912cd8c45d8c807aa10c041e4361b9a287ffd1d903c4fb6a3cf2733758208
Nov 03 12:50:39.407163 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:39.407134    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"]
Nov 03 12:50:40.115639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:40.115599    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d" event={"ID":"dddd777a-8170-415c-bc5e-47b844c113bb","Type":"ContainerStarted","Data":"61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e"}
Nov 03 12:50:40.116188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:40.115670    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:40.116188 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:40.115697    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d" event={"ID":"dddd777a-8170-415c-bc5e-47b844c113bb","Type":"ContainerStarted","Data":"5cc912cd8c45d8c807aa10c041e4361b9a287ffd1d903c4fb6a3cf2733758208"}
Nov 03 12:50:40.231887 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:40.231834    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d" podStartSLOduration=2.231818904 podStartE2EDuration="2.231818904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:50:40.221120394 +0000 UTC m=+3581.792704375" watchObservedRunningTime="2026-11-03 12:50:40.231818904 +0000 UTC m=+3581.803402815"
Nov 03 12:50:40.574331 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:40.574298    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df1a2fe-70e8-49de-abdc-ca9c1450dc7c" path="/var/lib/kubelet/pods/6df1a2fe-70e8-49de-abdc-ca9c1450dc7c/volumes"
Nov 03 12:50:51.126334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:51.126303    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:50:51.282485 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:51.282450    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"]
Nov 03 12:50:56.283426 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.283378    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn" podUID="d1914aac-6091-40b0-924d-a5fb9b13321c" containerName="payload-processing" containerID="cri-o://dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1" gracePeriod=25
Nov 03 12:50:56.546275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.546206    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:50:56.577044 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577019    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume\") pod \"d1914aac-6091-40b0-924d-a5fb9b13321c\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") "
Nov 03 12:50:56.577206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577084    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume\") pod \"d1914aac-6091-40b0-924d-a5fb9b13321c\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") "
Nov 03 12:50:56.577206 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577113    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87\") pod \"d1914aac-6091-40b0-924d-a5fb9b13321c\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") "
Nov 03 12:50:56.577346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577267    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials\") pod \"d1914aac-6091-40b0-924d-a5fb9b13321c\" (UID: \"d1914aac-6091-40b0-924d-a5fb9b13321c\") "
Nov 03 12:50:56.577408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577378    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume" pod "d1914aac-6091-40b0-924d-a5fb9b13321c" (UID: "d1914aac-6091-40b0-924d-a5fb9b13321c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:56.577469 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577408    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume" pod "d1914aac-6091-40b0-924d-a5fb9b13321c" (UID: "d1914aac-6091-40b0-924d-a5fb9b13321c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:50:56.577608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577590    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:56.577663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.577609    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1914aac-6091-40b0-924d-a5fb9b13321c-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:56.579493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.579472    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials" pod "d1914aac-6091-40b0-924d-a5fb9b13321c" (UID: "d1914aac-6091-40b0-924d-a5fb9b13321c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:56.579586 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.579539    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87" pod "d1914aac-6091-40b0-924d-a5fb9b13321c" (UID: "d1914aac-6091-40b0-924d-a5fb9b13321c"). InnerVolumeSpecName "kube-api-access-w4p87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:50:56.679170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.679099    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:56.679170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:56.679126    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4p87\" (UniqueName: \"kubernetes.io/projected/d1914aac-6091-40b0-924d-a5fb9b13321c-kube-api-access-w4p87\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:50:57.243972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.243944    4546 generic.go:356] "Generic (PLEG): container finished" podID="d1914aac-6091-40b0-924d-a5fb9b13321c" containerID="dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1" exitCode=0
Nov 03 12:50:57.244164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.243986    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"
Nov 03 12:50:57.244164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.244025    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn" event={"ID":"d1914aac-6091-40b0-924d-a5fb9b13321c","Type":"ContainerDied","Data":"dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1"}
Nov 03 12:50:57.244164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.244080    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn" event={"ID":"d1914aac-6091-40b0-924d-a5fb9b13321c","Type":"ContainerDied","Data":"3304588b7b9fc7bd88e8c4ef42abc7d0d9b22f44e8b06f7b7e461ccf94d8f519"}
Nov 03 12:50:57.244164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.244111    4546 scope.go:122] "RemoveContainer" containerID="dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1"
Nov 03 12:50:57.254659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.254636    4546 scope.go:122] "RemoveContainer" containerID="dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1"
Nov 03 12:50:57.254997 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:50:57.254976    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1\": container with ID starting with dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1 not found: ID does not exist" containerID="dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1"
Nov 03 12:50:57.255079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.255005    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1"} err="failed to get container status \"dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1\": rpc error: code = NotFound desc = could not find container \"dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1\": container with ID starting with dc2e60a2159915d45ec1fcf1c99455a1d0a8a3fdb4ead0f2f66ab8c45b56e3e1 not found: ID does not exist"
Nov 03 12:50:57.397096 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.397059    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"]
Nov 03 12:50:57.401763 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:57.401730    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-6dffvdltn"]
Nov 03 12:50:58.574558 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:50:58.574522    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1914aac-6091-40b0-924d-a5fb9b13321c" path="/var/lib/kubelet/pods/d1914aac-6091-40b0-924d-a5fb9b13321c/volumes"
Nov 03 12:51:07.652320 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:51:07.652278    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:51:37.651415 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:51:37.651329    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:51:43.470726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:43.470685    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"]
Nov 03 12:51:48.471364 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.471272    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4" podUID="bb7d3088-f7e9-414c-842c-8ff2d13c44e6" containerName="payload-processing" containerID="cri-o://b702e16b2e11954d235c80efbe06582112575f5c69e87b96b33bc404bb044d4e" gracePeriod=25
Nov 03 12:51:48.604110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.604045    4546 generic.go:356] "Generic (PLEG): container finished" podID="bb7d3088-f7e9-414c-842c-8ff2d13c44e6" containerID="b702e16b2e11954d235c80efbe06582112575f5c69e87b96b33bc404bb044d4e" exitCode=0
Nov 03 12:51:48.604235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.604112    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4" event={"ID":"bb7d3088-f7e9-414c-842c-8ff2d13c44e6","Type":"ContainerDied","Data":"b702e16b2e11954d235c80efbe06582112575f5c69e87b96b33bc404bb044d4e"}
Nov 03 12:51:48.736797 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.736773    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:51:48.859866 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.859785    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg\" (UniqueName: \"kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg\") pod \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") "
Nov 03 12:51:48.860052 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.859891    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume\") pod \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\" (UID: \"bb7d3088-f7e9-414c-842c-8ff2d13c44e6\") "
Nov 03 12:51:48.860343 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.860312    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume" pod "bb7d3088-f7e9-414c-842c-8ff2d13c44e6" (UID: "bb7d3088-f7e9-414c-842c-8ff2d13c44e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:51:48.860622 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.860602    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:48.862733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.862700    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg" pod "bb7d3088-f7e9-414c-842c-8ff2d13c44e6" (UID: "bb7d3088-f7e9-414c-842c-8ff2d13c44e6"). InnerVolumeSpecName "kube-api-access-z4qjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:48.962227 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:48.962183    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qjg\" (UniqueName: \"kubernetes.io/projected/bb7d3088-f7e9-414c-842c-8ff2d13c44e6-kube-api-access-z4qjg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:49.611833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:49.611803    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"
Nov 03 12:51:49.611833 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:49.611823    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4" event={"ID":"bb7d3088-f7e9-414c-842c-8ff2d13c44e6","Type":"ContainerDied","Data":"81363cbf13826f11ecfb13fe764de01c9185488a1bfd97407e8609fcfcfa9d20"}
Nov 03 12:51:49.612344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:49.611889    4546 scope.go:122] "RemoveContainer" containerID="b702e16b2e11954d235c80efbe06582112575f5c69e87b96b33bc404bb044d4e"
Nov 03 12:51:49.765027 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:49.764990    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"]
Nov 03 12:51:49.810849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:49.810806    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-aa20ad-7d6946c4f9-ftdp4"]
Nov 03 12:51:50.573267 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:50.573233    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7d3088-f7e9-414c-842c-8ff2d13c44e6" path="/var/lib/kubelet/pods/bb7d3088-f7e9-414c-842c-8ff2d13c44e6/volumes"
Nov 03 12:51:54.308506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:54.308464    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"]
Nov 03 12:51:54.309034 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:54.308728    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" containerID="cri-o://c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7" gracePeriod=30
Nov 03 12:51:57.513094 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:57.513058    4546 patch_prober.go:29] interesting pod/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:51:57.513580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:57.513120    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:51:59.569159 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.569100    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:51:59.673999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.673956    4546 generic.go:356] "Generic (PLEG): container finished" podID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerID="c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7" exitCode=0
Nov 03 12:51:59.674177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.674022    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" event={"ID":"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342","Type":"ContainerDied","Data":"c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7"}
Nov 03 12:51:59.674177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.674023    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"
Nov 03 12:51:59.674177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.674076    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr" event={"ID":"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342","Type":"ContainerDied","Data":"ece7771fec5c6ccd13379fe698d0e00b7b4f4b23c9a8e66ac58c06aaa8e6e08e"}
Nov 03 12:51:59.674177 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.674106    4546 scope.go:122] "RemoveContainer" containerID="c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7"
Nov 03 12:51:59.685525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.685505    4546 scope.go:122] "RemoveContainer" containerID="9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"
Nov 03 12:51:59.694289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.694269    4546 scope.go:122] "RemoveContainer" containerID="c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7"
Nov 03 12:51:59.694532 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:51:59.694513    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7\": container with ID starting with c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7 not found: ID does not exist" containerID="c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7"
Nov 03 12:51:59.694606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.694546    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7"} err="failed to get container status \"c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7\": rpc error: code = NotFound desc = could not find container \"c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7\": container with ID starting with c053e38a8cba27ce38d8e10b921855e49b23afec01adf51a8e7c550291305be7 not found: ID does not exist"
Nov 03 12:51:59.694606 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.694571    4546 scope.go:122] "RemoveContainer" containerID="9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"
Nov 03 12:51:59.694849 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:51:59.694828    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9\": container with ID starting with 9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9 not found: ID does not exist" containerID="9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"
Nov 03 12:51:59.694897 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.694858    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9"} err="failed to get container status \"9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9\": rpc error: code = NotFound desc = could not find container \"9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9\": container with ID starting with 9372b52f97183c23d87950cb24632b8aa504c375d2ac25672f9470f318765ed9 not found: ID does not exist"
Nov 03 12:51:59.704250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704232    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704292    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704317    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704341    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704398    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704423    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704508    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704525    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:59.704648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704600    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.704826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704696    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:59.704826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704707    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:59.704826 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704699    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket\") pod \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\" (UID: \"7b4c38e5-fc8e-40e4-ad27-bf4b3e260342\") "
Nov 03 12:51:59.705033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704824    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:51:59.705033 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.704984    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:59.705393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.705374    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.705479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.705398    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.705479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.705412    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.705479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.705426    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.705479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.705439    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.706731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.706704    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:51:59.706994 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.706966    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:51:59.707257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.707234    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "kube-api-access-hjfnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:59.707370 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.707266    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token" pod "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" (UID: "7b4c38e5-fc8e-40e4-ad27-bf4b3e260342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:59.806817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.806778    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.806817 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.806815    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjfnr\" (UniqueName: \"kubernetes.io/projected/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-kube-api-access-hjfnr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.807021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.806832    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:59.807021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:51:59.806845    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:00.133689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:00.133657    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"]
Nov 03 12:52:00.180395 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:00.180354    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fd7881-openshift-default-59c688c8f7-6gmdr"]
Nov 03 12:52:00.573664 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:00.573633    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" path="/var/lib/kubelet/pods/7b4c38e5-fc8e-40e4-ad27-bf4b3e260342/volumes"
Nov 03 12:52:00.709980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:00.709944    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"]
Nov 03 12:52:00.810037 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:00.810007    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"]
Nov 03 12:52:05.710523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:05.710465    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899" podUID="5b8deefe-bc86-4a39-ad4b-867441fb9a19" containerName="payload-processing" containerID="cri-o://8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf" gracePeriod=25
Nov 03 12:52:05.811068 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:05.811016    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62" podUID="e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" containerName="payload-pre-processing" containerID="cri-o://2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060" gracePeriod=25
Nov 03 12:52:05.973032 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:05.973010    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:52:06.096235 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.096203    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:52:06.106412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.106390    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms\" (UniqueName: \"kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms\") pod \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") "
Nov 03 12:52:06.106569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.106555    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume\") pod \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\" (UID: \"5b8deefe-bc86-4a39-ad4b-867441fb9a19\") "
Nov 03 12:52:06.106920 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.106895    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume" pod "5b8deefe-bc86-4a39-ad4b-867441fb9a19" (UID: "5b8deefe-bc86-4a39-ad4b-867441fb9a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:52:06.107069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.107051    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8deefe-bc86-4a39-ad4b-867441fb9a19-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:06.108791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.108766    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms" pod "5b8deefe-bc86-4a39-ad4b-867441fb9a19" (UID: "5b8deefe-bc86-4a39-ad4b-867441fb9a19"). InnerVolumeSpecName "kube-api-access-xg6ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:06.208361 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.208332    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg\" (UniqueName: \"kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg\") pod \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") "
Nov 03 12:52:06.208588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.208570    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume\") pod \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\" (UID: \"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba\") "
Nov 03 12:52:06.208938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.208906    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume" pod "e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" (UID: "e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:52:06.209249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.209087    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6ms\" (UniqueName: \"kubernetes.io/projected/5b8deefe-bc86-4a39-ad4b-867441fb9a19-kube-api-access-xg6ms\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:06.209249 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.209101    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:06.210726 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.210700    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg" pod "e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" (UID: "e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba"). InnerVolumeSpecName "kube-api-access-vfsdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:06.310582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.310501    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfsdg\" (UniqueName: \"kubernetes.io/projected/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba-kube-api-access-vfsdg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:06.723922 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.723895    4546 generic.go:356] "Generic (PLEG): container finished" podID="e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" containerID="2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060" exitCode=0
Nov 03 12:52:06.724371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.723947    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"
Nov 03 12:52:06.724371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.723976    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62" event={"ID":"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba","Type":"ContainerDied","Data":"2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060"}
Nov 03 12:52:06.724371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.724027    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62" event={"ID":"e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba","Type":"ContainerDied","Data":"d9a8d7ee93299aef2fd05701957aafe6e1978f10707ba08067e3b50a3e06a4b2"}
Nov 03 12:52:06.724371 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.724052    4546 scope.go:122] "RemoveContainer" containerID="2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060"
Nov 03 12:52:06.725608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.725588    4546 generic.go:356] "Generic (PLEG): container finished" podID="5b8deefe-bc86-4a39-ad4b-867441fb9a19" containerID="8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf" exitCode=0
Nov 03 12:52:06.725705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.725632    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"
Nov 03 12:52:06.725705 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.725654    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899" event={"ID":"5b8deefe-bc86-4a39-ad4b-867441fb9a19","Type":"ContainerDied","Data":"8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf"}
Nov 03 12:52:06.725812 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.725714    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899" event={"ID":"5b8deefe-bc86-4a39-ad4b-867441fb9a19","Type":"ContainerDied","Data":"8bb94413316d0de2fd61150c276e3ea1fd7928202a2cf32e7ac2bdbdf170fa66"}
Nov 03 12:52:06.737825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.737806    4546 scope.go:122] "RemoveContainer" containerID="2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060"
Nov 03 12:52:06.738109 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:52:06.738084    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060\": container with ID starting with 2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060 not found: ID does not exist" containerID="2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060"
Nov 03 12:52:06.738170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.738121    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060"} err="failed to get container status \"2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060\": rpc error: code = NotFound desc = could not find container \"2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060\": container with ID starting with 2c9d64fa55b73d1213d7bd9eb5a5d97f8279d8742ab0ecde46142a9de7eaa060 not found: ID does not exist"
Nov 03 12:52:06.738170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.738140    4546 scope.go:122] "RemoveContainer" containerID="8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf"
Nov 03 12:52:06.750587 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.750573    4546 scope.go:122] "RemoveContainer" containerID="8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf"
Nov 03 12:52:06.750907 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:52:06.750888    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf\": container with ID starting with 8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf not found: ID does not exist" containerID="8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf"
Nov 03 12:52:06.750984 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.750913    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf"} err="failed to get container status \"8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf\": rpc error: code = NotFound desc = could not find container \"8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf\": container with ID starting with 8e66ea29c6f47e8a2947bcd1251e7da6b5f791301fda543451407f3f761f64cf not found: ID does not exist"
Nov 03 12:52:06.878401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.878319    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"]
Nov 03 12:52:06.924211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:06.924180    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c84ae1-56c4ddb84f-qc899"]
Nov 03 12:52:07.026287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:07.026247    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"]
Nov 03 12:52:07.030262 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:07.030237    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c84ae1-7947c9f79f-zxp62"]
Nov 03 12:52:07.651249 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:52:07.651217    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:52:08.573643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:08.573603    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8deefe-bc86-4a39-ad4b-867441fb9a19" path="/var/lib/kubelet/pods/5b8deefe-bc86-4a39-ad4b-867441fb9a19/volumes"
Nov 03 12:52:08.574348 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:08.574326    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" path="/var/lib/kubelet/pods/e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba/volumes"
Nov 03 12:52:12.516528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.516494    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"]
Nov 03 12:52:12.517171 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517152    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d1914aac-6091-40b0-924d-a5fb9b13321c" containerName="payload-processing"
Nov 03 12:52:12.517219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517174    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1914aac-6091-40b0-924d-a5fb9b13321c" containerName="payload-processing"
Nov 03 12:52:12.517219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517191    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5b8deefe-bc86-4a39-ad4b-867441fb9a19" containerName="payload-processing"
Nov 03 12:52:12.517219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517196    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8deefe-bc86-4a39-ad4b-867441fb9a19" containerName="payload-processing"
Nov 03 12:52:12.517219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517212    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy"
Nov 03 12:52:12.517219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517216    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517228    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" containerName="payload-pre-processing"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517233    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" containerName="payload-pre-processing"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517248    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517254    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517268    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d3088-f7e9-414c-842c-8ff2d13c44e6" containerName="payload-processing"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517273    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d3088-f7e9-414c-842c-8ff2d13c44e6" containerName="payload-processing"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517368    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1914aac-6091-40b0-924d-a5fb9b13321c" containerName="payload-processing"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517379    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8deefe-bc86-4a39-ad4b-867441fb9a19" containerName="payload-processing"
Nov 03 12:52:12.517384 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517386    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy"
Nov 03 12:52:12.517674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517394    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e750a5cb-cfda-4b70-a4bf-ed374dc9a2ba" containerName="payload-pre-processing"
Nov 03 12:52:12.517674 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.517407    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d3088-f7e9-414c-842c-8ff2d13c44e6" containerName="payload-processing"
Nov 03 12:52:12.521681 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.521663    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.617232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.617174    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:52:12.617232 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.617182    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:52:12.617496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.617243    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:52:12.617496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.617271    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:52:12.620390 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.620366    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"]
Nov 03 12:52:12.685855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.685827    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.686095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.686072    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.686153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.686136    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdghx\" (UniqueName: \"kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.787805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.787708    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.787805 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.787770    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdghx\" (UniqueName: \"kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.788029 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.787838    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.788285 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.788267    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.790294 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.790275    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:12.841962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:12.841940    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdghx\" (UniqueName: \"kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx\") pod \"maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:13.135435 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:13.135369    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:13.501862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:13.501832    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"]
Nov 03 12:52:13.503252 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:52:13.503226    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35088af_5f0c_4863_a88f_5738fe5dadac.slice/crio-df6cb901bf631e3d5fe292f30a92cfb1895d669299630a1901de9f1bf7301a75 WatchSource:0}: Error finding container df6cb901bf631e3d5fe292f30a92cfb1895d669299630a1901de9f1bf7301a75: Status 404 returned error can't find the container with id df6cb901bf631e3d5fe292f30a92cfb1895d669299630a1901de9f1bf7301a75
Nov 03 12:52:13.771848 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:13.771782    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt" event={"ID":"f35088af-5f0c-4863-a88f-5738fe5dadac","Type":"ContainerStarted","Data":"df6cb901bf631e3d5fe292f30a92cfb1895d669299630a1901de9f1bf7301a75"}
Nov 03 12:52:14.779506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:14.779473    4546 generic.go:356] "Generic (PLEG): container finished" podID="f35088af-5f0c-4863-a88f-5738fe5dadac" containerID="0b7adbf56f5a3a777cbba4f16cbd35d806f304e3da31cd2c28288be9db1cab04" exitCode=0
Nov 03 12:52:14.779999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:14.779550    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt" event={"ID":"f35088af-5f0c-4863-a88f-5738fe5dadac","Type":"ContainerDied","Data":"0b7adbf56f5a3a777cbba4f16cbd35d806f304e3da31cd2c28288be9db1cab04"}
Nov 03 12:52:15.939472 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:15.939448    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:16.026393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.026355    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca\") pod \"f35088af-5f0c-4863-a88f-5738fe5dadac\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") "
Nov 03 12:52:16.026569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.026428    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx\" (UniqueName: \"kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx\") pod \"f35088af-5f0c-4863-a88f-5738fe5dadac\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") "
Nov 03 12:52:16.026569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.026472    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca\") pod \"f35088af-5f0c-4863-a88f-5738fe5dadac\" (UID: \"f35088af-5f0c-4863-a88f-5738fe5dadac\") "
Nov 03 12:52:16.027457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.027416    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca" pod "f35088af-5f0c-4863-a88f-5738fe5dadac" (UID: "f35088af-5f0c-4863-a88f-5738fe5dadac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:52:16.028929 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.028900    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx" pod "f35088af-5f0c-4863-a88f-5738fe5dadac" (UID: "f35088af-5f0c-4863-a88f-5738fe5dadac"). InnerVolumeSpecName "kube-api-access-zdghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:16.029039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.028975    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca" pod "f35088af-5f0c-4863-a88f-5738fe5dadac" (UID: "f35088af-5f0c-4863-a88f-5738fe5dadac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:52:16.128243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.128212    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:16.128243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.128238    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdghx\" (UniqueName: \"kubernetes.io/projected/f35088af-5f0c-4863-a88f-5738fe5dadac-kube-api-access-zdghx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:16.128243 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.128249    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35088af-5f0c-4863-a88f-5738fe5dadac-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:16.793228 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.793202    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"
Nov 03 12:52:16.793403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.793245    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt" event={"ID":"f35088af-5f0c-4863-a88f-5738fe5dadac","Type":"ContainerDied","Data":"df6cb901bf631e3d5fe292f30a92cfb1895d669299630a1901de9f1bf7301a75"}
Nov 03 12:52:16.793403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:16.793292    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6cb901bf631e3d5fe292f30a92cfb1895d669299630a1901de9f1bf7301a75"
Nov 03 12:52:37.651923 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:52:37.651884    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:52:51.343400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.343325    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"]
Nov 03 12:52:51.343996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.343978    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f35088af-5f0c-4863-a88f-5738fe5dadac" containerName="revoke-keys"
Nov 03 12:52:51.343996 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.343996    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35088af-5f0c-4863-a88f-5738fe5dadac" containerName="revoke-keys"
Nov 03 12:52:51.344145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.344123    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4c38e5-fc8e-40e4-ad27-bf4b3e260342" containerName="istio-proxy"
Nov 03 12:52:51.344145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.344135    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35088af-5f0c-4863-a88f-5738fe5dadac" containerName="revoke-keys"
Nov 03 12:52:51.347879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.347862    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.401668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.401642    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"]
Nov 03 12:52:51.443919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.443893    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:52:51.443919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.443913    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:52:51.443919 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.443908    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:52:51.444147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.443973    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:52:51.481210 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.481181    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z665\" (UniqueName: \"kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.481369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.481352    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.481413 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.481385    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.582759 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.582725    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.582926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.582793    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.582926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.582881    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z665\" (UniqueName: \"kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.583346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.583325    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.585124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.585102    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.635248 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.635192    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z665\" (UniqueName: \"kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665\") pod \"maas-api-revoke-sub-5f495b88-q98j5\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:51.661528 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:51.661504    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:52.027203 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:52:52.027156    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd808ede0_6792_4ba9_90f9_8026162e1404.slice/crio-a9bd5f418b73d2bf9f277bce54a261d520d564efda6594aa8806e78d706bde40 WatchSource:0}: Error finding container a9bd5f418b73d2bf9f277bce54a261d520d564efda6594aa8806e78d706bde40: Status 404 returned error can't find the container with id a9bd5f418b73d2bf9f277bce54a261d520d564efda6594aa8806e78d706bde40
Nov 03 12:52:52.028222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:52.027896    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"]
Nov 03 12:52:53.018795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:53.018759    4546 generic.go:356] "Generic (PLEG): container finished" podID="d808ede0-6792-4ba9-90f9-8026162e1404" containerID="7b58b64b5e4d0298155a8189cc2a47bc780406f3a506e089d771b966984557ec" exitCode=0
Nov 03 12:52:53.019169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:53.018800    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5" event={"ID":"d808ede0-6792-4ba9-90f9-8026162e1404","Type":"ContainerDied","Data":"7b58b64b5e4d0298155a8189cc2a47bc780406f3a506e089d771b966984557ec"}
Nov 03 12:52:53.019169 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:53.018867    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5" event={"ID":"d808ede0-6792-4ba9-90f9-8026162e1404","Type":"ContainerStarted","Data":"a9bd5f418b73d2bf9f277bce54a261d520d564efda6594aa8806e78d706bde40"}
Nov 03 12:52:54.200453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.200425    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:54.315646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.315560    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca\") pod \"d808ede0-6792-4ba9-90f9-8026162e1404\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") "
Nov 03 12:52:54.315849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.315685    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665\" (UniqueName: \"kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665\") pod \"d808ede0-6792-4ba9-90f9-8026162e1404\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") "
Nov 03 12:52:54.315935 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.315846    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca\") pod \"d808ede0-6792-4ba9-90f9-8026162e1404\" (UID: \"d808ede0-6792-4ba9-90f9-8026162e1404\") "
Nov 03 12:52:54.316425 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.316393    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca" pod "d808ede0-6792-4ba9-90f9-8026162e1404" (UID: "d808ede0-6792-4ba9-90f9-8026162e1404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:52:54.318583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.318555    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665" pod "d808ede0-6792-4ba9-90f9-8026162e1404" (UID: "d808ede0-6792-4ba9-90f9-8026162e1404"). InnerVolumeSpecName "kube-api-access-5z665". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:54.318865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.318838    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca" pod "d808ede0-6792-4ba9-90f9-8026162e1404" (UID: "d808ede0-6792-4ba9-90f9-8026162e1404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:52:54.417506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.417459    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:54.417506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.417497    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d808ede0-6792-4ba9-90f9-8026162e1404-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:54.417506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:54.417513    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z665\" (UniqueName: \"kubernetes.io/projected/d808ede0-6792-4ba9-90f9-8026162e1404-kube-api-access-5z665\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:55.033104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:55.033076    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"
Nov 03 12:52:55.033287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:55.033124    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5" event={"ID":"d808ede0-6792-4ba9-90f9-8026162e1404","Type":"ContainerDied","Data":"a9bd5f418b73d2bf9f277bce54a261d520d564efda6594aa8806e78d706bde40"}
Nov 03 12:52:55.033287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:52:55.033170    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9bd5f418b73d2bf9f277bce54a261d520d564efda6594aa8806e78d706bde40"
Nov 03 12:53:02.048393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.048359    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"]
Nov 03 12:53:02.049022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.049006    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d808ede0-6792-4ba9-90f9-8026162e1404" containerName="revoke-keys"
Nov 03 12:53:02.049022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.049022    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d808ede0-6792-4ba9-90f9-8026162e1404" containerName="revoke-keys"
Nov 03 12:53:02.049138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.049132    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d808ede0-6792-4ba9-90f9-8026162e1404" containerName="revoke-keys"
Nov 03 12:53:02.059623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.059586    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.105408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.105370    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"]
Nov 03 12:53:02.147766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.147724    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:53:02.147766 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.147733    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:53:02.147983 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.147738    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:53:02.154213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.154193    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:53:02.230153 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.230106    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.230321 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.230305    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.230519 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.230494    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkk2s\" (UniqueName: \"kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.331556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.331465    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkk2s\" (UniqueName: \"kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.331556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.331519    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.331857 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.331611    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.332272 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.332251    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.334062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.334030    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.393211 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.393180    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkk2s\" (UniqueName: \"kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s\") pod \"maas-api-revoke-sub-3446e3d6-7rqpt\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:02.673629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:02.673542    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:03.040130 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:03.040101    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"]
Nov 03 12:53:03.041641 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:53:03.041605    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a1bebf_5b9b_4940_99a3_e03edc4aa3b1.slice/crio-d80d44dacbfcff257ce518ce7bc00402b4b4972cd64b716f63eb77825f82a996 WatchSource:0}: Error finding container d80d44dacbfcff257ce518ce7bc00402b4b4972cd64b716f63eb77825f82a996: Status 404 returned error can't find the container with id d80d44dacbfcff257ce518ce7bc00402b4b4972cd64b716f63eb77825f82a996
Nov 03 12:53:03.114356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:03.114318    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt" event={"ID":"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1","Type":"ContainerStarted","Data":"d80d44dacbfcff257ce518ce7bc00402b4b4972cd64b716f63eb77825f82a996"}
Nov 03 12:53:04.135772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:04.132151    4546 generic.go:356] "Generic (PLEG): container finished" podID="07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" containerID="9701b77492324e519c36d8b5f07cd22fa3d0c90e1b24c41e8b1836fb01227037" exitCode=0
Nov 03 12:53:04.135772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:04.132201    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt" event={"ID":"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1","Type":"ContainerDied","Data":"9701b77492324e519c36d8b5f07cd22fa3d0c90e1b24c41e8b1836fb01227037"}
Nov 03 12:53:05.296049 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.296025    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:05.379125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.379085    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca\") pod \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") "
Nov 03 12:53:05.379125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.379125    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s\" (UniqueName: \"kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s\") pod \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") "
Nov 03 12:53:05.379365 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.379210    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca\") pod \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\" (UID: \"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1\") "
Nov 03 12:53:05.379542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.379521    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca" pod "07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" (UID: "07a1bebf-5b9b-4940-99a3-e03edc4aa3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:05.379680 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.379664    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:05.381418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.381392    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca" pod "07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" (UID: "07a1bebf-5b9b-4940-99a3-e03edc4aa3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:53:05.381530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.381501    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s" pod "07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" (UID: "07a1bebf-5b9b-4940-99a3-e03edc4aa3b1"). InnerVolumeSpecName "kube-api-access-hkk2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:05.481323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.481284    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkk2s\" (UniqueName: \"kubernetes.io/projected/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-kube-api-access-hkk2s\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:05.481323 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:05.481320    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:06.145899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:06.145858    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"
Nov 03 12:53:06.145899 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:06.145867    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt" event={"ID":"07a1bebf-5b9b-4940-99a3-e03edc4aa3b1","Type":"ContainerDied","Data":"d80d44dacbfcff257ce518ce7bc00402b4b4972cd64b716f63eb77825f82a996"}
Nov 03 12:53:06.146191 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:06.145911    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80d44dacbfcff257ce518ce7bc00402b4b4972cd64b716f63eb77825f82a996"
Nov 03 12:53:07.655507 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:53:07.655427    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:53:26.693268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.693234    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"]
Nov 03 12:53:26.693835 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.693820    4546 cpu_manager.go:414] "containerMap: removing container" podUID="07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" containerName="revoke-keys"
Nov 03 12:53:26.693890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.693836    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" containerName="revoke-keys"
Nov 03 12:53:26.693945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.693936    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" containerName="revoke-keys"
Nov 03 12:53:26.697761 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.697729    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.747850 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.747827    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6e306ea651ee\""
Nov 03 12:53:26.751209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.751187    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"]
Nov 03 12:53:26.774640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.774605    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqsm\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.774640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.774641    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.774839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.774791    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.774878 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.774848    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.793923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.793897    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1ab9c770bc22\""
Nov 03 12:53:26.793923 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.793903    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:53:26.794081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.793947    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:53:26.794081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.793953    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6lkbc\""
Nov 03 12:53:26.794081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.793982    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:53:26.794081 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.793981    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:53:26.875665 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.875643    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqsm\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.875820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.875670    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.875820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.875782    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.875938 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.875851    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.876429 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.876407    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.876518 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.876463    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.878213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.878195    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:26.928346 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:26.928319    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqsm\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm\") pod \"payload-processing-external-model-5fb455ff55-pn5ph\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") " pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:27.010915 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:27.010888    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:27.400526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:27.399406    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"]
Nov 03 12:53:27.405464 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:53:27.405425    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f0b095_4ba2_44f6_8801_de727386a6ab.slice/crio-1332cff031b9d8a114cfd6de7fd07ec2b35df332a35c583bc8a4abb627ba31e5 WatchSource:0}: Error finding container 1332cff031b9d8a114cfd6de7fd07ec2b35df332a35c583bc8a4abb627ba31e5: Status 404 returned error can't find the container with id 1332cff031b9d8a114cfd6de7fd07ec2b35df332a35c583bc8a4abb627ba31e5
Nov 03 12:53:28.290720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:28.290686    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph" event={"ID":"f2f0b095-4ba2-44f6-8801-de727386a6ab","Type":"ContainerStarted","Data":"8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965"}
Nov 03 12:53:28.291147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:28.290741    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph" event={"ID":"f2f0b095-4ba2-44f6-8801-de727386a6ab","Type":"ContainerStarted","Data":"1332cff031b9d8a114cfd6de7fd07ec2b35df332a35c583bc8a4abb627ba31e5"}
Nov 03 12:53:28.291147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:28.290810    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:28.398777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:28.398712    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph" podStartSLOduration=2.398698934 podStartE2EDuration="2.398698934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:28.395246767 +0000 UTC m=+3749.966830692" watchObservedRunningTime="2026-11-03 12:53:28.398698934 +0000 UTC m=+3749.970282867"
Nov 03 12:53:28.856550 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:28.853902    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"]
Nov 03 12:53:28.858481 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:28.858452    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-38f178-mwqtt"]
Nov 03 12:53:29.084739 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:29.084710    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"]
Nov 03 12:53:29.085219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:29.085191    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy" containerID="cri-o://b2fb29d4b204d5a11992145403917dd3a3e98f057a426d077f12cc71f2731aa0" gracePeriod=30
Nov 03 12:53:30.576870 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:30.576840    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35088af-5f0c-4863-a88f-5738fe5dadac" path="/var/lib/kubelet/pods/f35088af-5f0c-4863-a88f-5738fe5dadac/volumes"
Nov 03 12:53:34.338450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.338420    4546 generic.go:356] "Generic (PLEG): container finished" podID="3141d97b-c167-49c0-b1e6-2596452f6117" containerID="b2fb29d4b204d5a11992145403917dd3a3e98f057a426d077f12cc71f2731aa0" exitCode=0
Nov 03 12:53:34.338917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.338502    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" event={"ID":"3141d97b-c167-49c0-b1e6-2596452f6117","Type":"ContainerDied","Data":"b2fb29d4b204d5a11992145403917dd3a3e98f057a426d077f12cc71f2731aa0"}
Nov 03 12:53:34.338917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.338574    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2" event={"ID":"3141d97b-c167-49c0-b1e6-2596452f6117","Type":"ContainerDied","Data":"c6ff2b3334cc250e2f0097bbe9d4ab22490bb1aa1f480de49e6ac4808e88d16e"}
Nov 03 12:53:34.338917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.338605    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ff2b3334cc250e2f0097bbe9d4ab22490bb1aa1f480de49e6ac4808e88d16e"
Nov 03 12:53:34.338917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.338622    4546 scope.go:122] "RemoveContainer" containerID="e37ab13ea39c3e41aedbaf51559efa7a54038307a722eebc231ebd2f143e7ff0"
Nov 03 12:53:34.359486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.359466    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:53:34.487076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487044    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487145    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487186    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487263 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487212    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487270    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487301    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487340    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487301    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:34.487650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487473    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:34.487650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487478    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487684    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket\") pod \"3141d97b-c167-49c0-b1e6-2596452f6117\" (UID: \"3141d97b-c167-49c0-b1e6-2596452f6117\") "
Nov 03 12:53:34.487793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487704    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:34.487917 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487822    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:34.487974 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.487946    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:34.488363 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.488338    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.488454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.488368    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.488454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.488384    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.488454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.488399    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.488454 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.488413    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3141d97b-c167-49c0-b1e6-2596452f6117-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.489648 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.489620    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:53:34.489784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.489653    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:34.489784 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.489698    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:34.489896 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.489809    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb" pod "3141d97b-c167-49c0-b1e6-2596452f6117" (UID: "3141d97b-c167-49c0-b1e6-2596452f6117"). InnerVolumeSpecName "kube-api-access-bjdfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:34.590367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.590328    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdfb\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-kube-api-access-bjdfb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.590367 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.590370    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3141d97b-c167-49c0-b1e6-2596452f6117-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.590577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.590384    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3141d97b-c167-49c0-b1e6-2596452f6117-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:34.590577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:34.590403    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3141d97b-c167-49c0-b1e6-2596452f6117-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:35.346772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:35.346732    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"
Nov 03 12:53:35.606345 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:35.606263    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"]
Nov 03 12:53:35.652631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:35.652602    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-38f178-openshift-default-76b8f7c84f-m4fl2"]
Nov 03 12:53:36.122134 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:36.122104    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"]
Nov 03 12:53:36.124966 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:36.124943    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:36.573856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:36.573828    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" path="/var/lib/kubelet/pods/3141d97b-c167-49c0-b1e6-2596452f6117/volumes"
Nov 03 12:53:37.650800 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:53:37.650766    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:53:41.123238 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.123196    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph" podUID="f2f0b095-4ba2-44f6-8801-de727386a6ab" containerName="payload-processing" containerID="cri-o://8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965" gracePeriod=25
Nov 03 12:53:41.374458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.374397    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:41.382842 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.382817    4546 generic.go:356] "Generic (PLEG): container finished" podID="f2f0b095-4ba2-44f6-8801-de727386a6ab" containerID="8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965" exitCode=0
Nov 03 12:53:41.382958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.382860    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"
Nov 03 12:53:41.382958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.382897    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph" event={"ID":"f2f0b095-4ba2-44f6-8801-de727386a6ab","Type":"ContainerDied","Data":"8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965"}
Nov 03 12:53:41.382958 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.382950    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph" event={"ID":"f2f0b095-4ba2-44f6-8801-de727386a6ab","Type":"ContainerDied","Data":"1332cff031b9d8a114cfd6de7fd07ec2b35df332a35c583bc8a4abb627ba31e5"}
Nov 03 12:53:41.383075 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.382973    4546 scope.go:122] "RemoveContainer" containerID="8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965"
Nov 03 12:53:41.391666 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.391648    4546 scope.go:122] "RemoveContainer" containerID="8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965"
Nov 03 12:53:41.391967 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:53:41.391948    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965\": container with ID starting with 8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965 not found: ID does not exist" containerID="8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965"
Nov 03 12:53:41.392058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.391975    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965"} err="failed to get container status \"8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965\": rpc error: code = NotFound desc = could not find container \"8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965\": container with ID starting with 8fb0986d148aa16f32f342ee3fefdcfa9b7b25035d3267cca332207e32df3965 not found: ID does not exist"
Nov 03 12:53:41.463729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.463706    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume\") pod \"f2f0b095-4ba2-44f6-8801-de727386a6ab\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") "
Nov 03 12:53:41.463882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.463838    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume\") pod \"f2f0b095-4ba2-44f6-8801-de727386a6ab\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") "
Nov 03 12:53:41.463882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.463869    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm\") pod \"f2f0b095-4ba2-44f6-8801-de727386a6ab\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") "
Nov 03 12:53:41.463999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.463986    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials\") pod \"f2f0b095-4ba2-44f6-8801-de727386a6ab\" (UID: \"f2f0b095-4ba2-44f6-8801-de727386a6ab\") "
Nov 03 12:53:41.464091 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.464021    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume" pod "f2f0b095-4ba2-44f6-8801-de727386a6ab" (UID: "f2f0b095-4ba2-44f6-8801-de727386a6ab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:41.464205 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.464185    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume" pod "f2f0b095-4ba2-44f6-8801-de727386a6ab" (UID: "f2f0b095-4ba2-44f6-8801-de727386a6ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:41.464397 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.464383    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:41.464466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.464398    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2f0b095-4ba2-44f6-8801-de727386a6ab-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:41.466132 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.466102    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm" pod "f2f0b095-4ba2-44f6-8801-de727386a6ab" (UID: "f2f0b095-4ba2-44f6-8801-de727386a6ab"). InnerVolumeSpecName "kube-api-access-mfqsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:41.466225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.466132    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials" pod "f2f0b095-4ba2-44f6-8801-de727386a6ab" (UID: "f2f0b095-4ba2-44f6-8801-de727386a6ab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:41.565608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.565576    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:41.565608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.565603    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqsm\" (UniqueName: \"kubernetes.io/projected/f2f0b095-4ba2-44f6-8801-de727386a6ab-kube-api-access-mfqsm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:41.795144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.795109    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"]
Nov 03 12:53:41.800125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:41.800098    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb455ff55-pn5ph"]
Nov 03 12:53:42.574475 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:42.574432    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f0b095-4ba2-44f6-8801-de727386a6ab" path="/var/lib/kubelet/pods/f2f0b095-4ba2-44f6-8801-de727386a6ab/volumes"
Nov 03 12:53:55.122659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.122626    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"]
Nov 03 12:53:55.123239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123224    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f2f0b095-4ba2-44f6-8801-de727386a6ab" containerName="payload-processing"
Nov 03 12:53:55.123302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123241    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f0b095-4ba2-44f6-8801-de727386a6ab" containerName="payload-processing"
Nov 03 12:53:55.123302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123256    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy"
Nov 03 12:53:55.123302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123261    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy"
Nov 03 12:53:55.123302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123276    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy"
Nov 03 12:53:55.123302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123281    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy"
Nov 03 12:53:55.123479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123362    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f0b095-4ba2-44f6-8801-de727386a6ab" containerName="payload-processing"
Nov 03 12:53:55.123479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.123373    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy"
Nov 03 12:53:55.128289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.128270    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221136    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221267    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjgn\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221312    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221330    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221532 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221510    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221625 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221612    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221636    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221673 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221666    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.221779 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.221765    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324047    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324124    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324142    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324170    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324199    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324260 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324259    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324302    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjgn\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324350    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.324597 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.324378    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.325110 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.325086    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.325276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.325116    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.325505 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.325483    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.325631 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.325613    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.325933 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.325906    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.327645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.327618    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.327993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.327969    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.328275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.328251    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"]
Nov 03 12:53:55.377510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.377441    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjgn\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.377600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.377583    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xrrqv\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.441219 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.441194    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:55.853939 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.853913    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"]
Nov 03 12:53:55.855258 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:53:55.855229    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda752bfa7_4293_4c5a_b089_a8c1fd37bd41.slice/crio-5bd626ba78a45ca016cb2bbe3cda1b8e66e2c8565f8c309b815abaf820651b7d WatchSource:0}: Error finding container 5bd626ba78a45ca016cb2bbe3cda1b8e66e2c8565f8c309b815abaf820651b7d: Status 404 returned error can't find the container with id 5bd626ba78a45ca016cb2bbe3cda1b8e66e2c8565f8c309b815abaf820651b7d
Nov 03 12:53:55.857556 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.857523    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:53:55.857620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.857590    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:53:55.857663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:55.857621    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:53:56.483611 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:56.483574    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" event={"ID":"a752bfa7-4293-4c5a-b089-a8c1fd37bd41","Type":"ContainerStarted","Data":"46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14"}
Nov 03 12:53:56.484039 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:56.483639    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" event={"ID":"a752bfa7-4293-4c5a-b089-a8c1fd37bd41","Type":"ContainerStarted","Data":"5bd626ba78a45ca016cb2bbe3cda1b8e66e2c8565f8c309b815abaf820651b7d"}
Nov 03 12:53:56.686353 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:56.686307    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podStartSLOduration=1.6862935989999999 podStartE2EDuration="1.686293599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:56.637553154 +0000 UTC m=+3778.209137090" watchObservedRunningTime="2026-11-03 12:53:56.686293599 +0000 UTC m=+3778.257877509"
Nov 03 12:53:57.442330 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:57.442288    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:53:57.443882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:57.443861    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xrrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 12:53:57.443964 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:57.443909    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 12:53:58.441642 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:58.441587    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xrrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 12:53:58.442138 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:58.441661    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 12:53:59.442336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:59.442297    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xrrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 12:53:59.442733 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:53:59.442362    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 12:54:00.442549 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.442509    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xrrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 12:54:00.443089 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.442572    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 12:54:00.968146 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.968111    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"]
Nov 03 12:54:00.968926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.968897    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3141d97b-c167-49c0-b1e6-2596452f6117" containerName="istio-proxy"
Nov 03 12:54:00.972496 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.972482    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:00.997690 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.997666    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:00.997820 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.997713    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc6f\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:00.997951 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.997934    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:00.997990 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:00.997962    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.077104 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077073    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9ec01e4d84c9\""
Nov 03 12:54:01.077310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077081    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:54:01.077310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077090    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1ab9c770bc22\""
Nov 03 12:54:01.077310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077090    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-98537d3e6233\""
Nov 03 12:54:01.077310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077094    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6e306ea651ee\""
Nov 03 12:54:01.077310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077142    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:54:01.077310 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077277    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:54:01.077560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077332    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:54:01.077560 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.077378    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6lkbc\""
Nov 03 12:54:01.081967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.081942    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"]
Nov 03 12:54:01.099250 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.099224    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.099400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.099258    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.099400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.099329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.099400 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.099365    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc6f\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.100069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.100044    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.100145 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.100137    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.102036 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.102017    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.152582 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.152544    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc6f\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f\") pod \"payload-processing-external-model-5f469f87f8-h4j7f\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") " pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.286296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.286209    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:01.442507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.442468    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xrrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 12:54:01.442689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.442537    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 12:54:01.678322 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:01.678295    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"]
Nov 03 12:54:01.679332 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:54:01.679302    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5642dd97_f760_4fe5_b11f_a7aecb0c0487.slice/crio-913c4e248b6afc60059073a28cb2e2c318881f1f331cba4dba22118791e3e7fa WatchSource:0}: Error finding container 913c4e248b6afc60059073a28cb2e2c318881f1f331cba4dba22118791e3e7fa: Status 404 returned error can't find the container with id 913c4e248b6afc60059073a28cb2e2c318881f1f331cba4dba22118791e3e7fa
Nov 03 12:54:02.441626 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:02.441590    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xrrqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 12:54:02.441856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:02.441641    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 12:54:02.545538 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:02.545492    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f" event={"ID":"5642dd97-f760-4fe5-b11f-a7aecb0c0487","Type":"ContainerStarted","Data":"2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2"}
Nov 03 12:54:02.546087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:02.545580    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f" event={"ID":"5642dd97-f760-4fe5-b11f-a7aecb0c0487","Type":"ContainerStarted","Data":"913c4e248b6afc60059073a28cb2e2c318881f1f331cba4dba22118791e3e7fa"}
Nov 03 12:54:02.546087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:02.545604    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:02.653798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:02.653725    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f" podStartSLOduration=2.653711268 podStartE2EDuration="2.653711268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:02.651396298 +0000 UTC m=+3784.222980227" watchObservedRunningTime="2026-11-03 12:54:02.653711268 +0000 UTC m=+3784.225295178"
Nov 03 12:54:03.633212 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:03.633180    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-98537d3e6233: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:03.633212 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:03.633210    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:03.633800 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:03.633270    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials podName:5642dd97-f760-4fe5-b11f-a7aecb0c0487 nodeName:}" failed. No retries permitted until 2026-11-03 12:54:04.133248993 +0000 UTC m=+3785.704832898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials") pod "payload-processing-external-model-5f469f87f8-h4j7f" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487") : secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:04.140715 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:04.140675    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-98537d3e6233: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:04.140715 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:04.140712    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:04.141042 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:04.140824    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials podName:5642dd97-f760-4fe5-b11f-a7aecb0c0487 nodeName:}" failed. No retries permitted until 2026-11-03 12:54:05.140804423 +0000 UTC m=+3786.712388325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials") pod "payload-processing-external-model-5f469f87f8-h4j7f" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487") : secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:04.271086 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:04.271058    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:54:04.271580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:04.271565    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:54:04.272463 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:04.272445    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 12:54:05.175545 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:05.175504    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-98537d3e6233: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:05.175545 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:05.175534    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:05.176070 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:05.175595    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials podName:5642dd97-f760-4fe5-b11f-a7aecb0c0487 nodeName:}" failed. No retries permitted until 2026-11-03 12:54:07.175575139 +0000 UTC m=+3788.747159028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials") pod "payload-processing-external-model-5f469f87f8-h4j7f" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487") : secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:05.668764 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:05.668711    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"]
Nov 03 12:54:07.201146 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:07.201115    4546 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-98537d3e6233: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:07.201146 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:07.201141    4546 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f: secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:07.201596 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:07.201202    4546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials podName:5642dd97-f760-4fe5-b11f-a7aecb0c0487 nodeName:}" failed. No retries permitted until 2026-11-03 12:54:11.201189759 +0000 UTC m=+3792.772773660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials") pod "payload-processing-external-model-5f469f87f8-h4j7f" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487") : secret "agc-cred-98537d3e6233" not found
Nov 03 12:54:07.653282 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:07.653245    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:54:10.669398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:10.669364    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f" podUID="5642dd97-f760-4fe5-b11f-a7aecb0c0487" containerName="payload-processing" containerID="cri-o://2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2" gracePeriod=25
Nov 03 12:54:10.940829 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:10.940629    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:11.046843 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.046807    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume\") pod \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") "
Nov 03 12:54:11.047015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.046897    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume\") pod \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") "
Nov 03 12:54:11.047073 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.047034    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f\") pod \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") "
Nov 03 12:54:11.047136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.047087    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials\") pod \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\" (UID: \"5642dd97-f760-4fe5-b11f-a7aecb0c0487\") "
Nov 03 12:54:11.047203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.047180    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume" pod "5642dd97-f760-4fe5-b11f-a7aecb0c0487" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:54:11.047314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.047287    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume" pod "5642dd97-f760-4fe5-b11f-a7aecb0c0487" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:54:11.047508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.047489    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:11.047508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.047505    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5642dd97-f760-4fe5-b11f-a7aecb0c0487-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:11.049507 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.049484    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f" pod "5642dd97-f760-4fe5-b11f-a7aecb0c0487" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487"). InnerVolumeSpecName "kube-api-access-vfc6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:54:11.049612 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.049584    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials" pod "5642dd97-f760-4fe5-b11f-a7aecb0c0487" (UID: "5642dd97-f760-4fe5-b11f-a7aecb0c0487"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:54:11.149247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.149219    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:11.149247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.149243    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfc6f\" (UniqueName: \"kubernetes.io/projected/5642dd97-f760-4fe5-b11f-a7aecb0c0487-kube-api-access-vfc6f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:11.605980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.605949    4546 generic.go:356] "Generic (PLEG): container finished" podID="5642dd97-f760-4fe5-b11f-a7aecb0c0487" containerID="2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2" exitCode=0
Nov 03 12:54:11.606187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.606015    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"
Nov 03 12:54:11.606187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.606031    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f" event={"ID":"5642dd97-f760-4fe5-b11f-a7aecb0c0487","Type":"ContainerDied","Data":"2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2"}
Nov 03 12:54:11.606187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.606096    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f" event={"ID":"5642dd97-f760-4fe5-b11f-a7aecb0c0487","Type":"ContainerDied","Data":"913c4e248b6afc60059073a28cb2e2c318881f1f331cba4dba22118791e3e7fa"}
Nov 03 12:54:11.606187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.606117    4546 scope.go:122] "RemoveContainer" containerID="2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2"
Nov 03 12:54:11.646600 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.644263    4546 scope.go:122] "RemoveContainer" containerID="2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2"
Nov 03 12:54:11.646834 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:11.646685    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2\": container with ID starting with 2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2 not found: ID does not exist" containerID="2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2"
Nov 03 12:54:11.646834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.646723    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2"} err="failed to get container status \"2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2\": rpc error: code = NotFound desc = could not find container \"2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2\": container with ID starting with 2796b7e9232e7a13168b178cad8ab88e925ffcc4f0e78b685fcf2a399899dac2 not found: ID does not exist"
Nov 03 12:54:11.759282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.759245    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"]
Nov 03 12:54:11.805433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:11.805404    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f469f87f8-h4j7f"]
Nov 03 12:54:12.573021 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:12.572990    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5642dd97-f760-4fe5-b11f-a7aecb0c0487" path="/var/lib/kubelet/pods/5642dd97-f760-4fe5-b11f-a7aecb0c0487/volumes"
Nov 03 12:54:15.500026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.499987    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"]
Nov 03 12:54:15.500548 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.500533    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5642dd97-f760-4fe5-b11f-a7aecb0c0487" containerName="payload-processing"
Nov 03 12:54:15.500594 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.500548    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5642dd97-f760-4fe5-b11f-a7aecb0c0487" containerName="payload-processing"
Nov 03 12:54:15.500741 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.500727    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5642dd97-f760-4fe5-b11f-a7aecb0c0487" containerName="payload-processing"
Nov 03 12:54:15.506088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.506066    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.608769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.607777    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"]
Nov 03 12:54:15.630772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.620066    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b737fe-dockercfg-tsjg7\""
Nov 03 12:54:15.630772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.620136    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b737fe\""
Nov 03 12:54:15.630772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.621565    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.630772 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.621710    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6bs\" (UniqueName: \"kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs\") pod \"payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.752048 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.751965    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.752216 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.752088    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6bs\" (UniqueName: \"kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs\") pod \"payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.752906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.752882    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.805088 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.805038    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6bs\" (UniqueName: \"kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs\") pod \"payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:15.820736 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:15.820706    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:16.192978 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:16.192949    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"]
Nov 03 12:54:16.194105 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:54:16.194061    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e71e1f9_e133_41ad_90a8_52bfffc343db.slice/crio-740eb33550d0540f18b86f802012171a5b0575808a4a42e5fcd7c370a838f172 WatchSource:0}: Error finding container 740eb33550d0540f18b86f802012171a5b0575808a4a42e5fcd7c370a838f172: Status 404 returned error can't find the container with id 740eb33550d0540f18b86f802012171a5b0575808a4a42e5fcd7c370a838f172
Nov 03 12:54:16.662912 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:16.662875    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9" event={"ID":"6e71e1f9-e133-41ad-90a8-52bfffc343db","Type":"ContainerStarted","Data":"8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da"}
Nov 03 12:54:16.663325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:16.662941    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:16.663325 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:16.662965    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9" event={"ID":"6e71e1f9-e133-41ad-90a8-52bfffc343db","Type":"ContainerStarted","Data":"740eb33550d0540f18b86f802012171a5b0575808a4a42e5fcd7c370a838f172"}
Nov 03 12:54:16.769062 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:16.769011    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9" podStartSLOduration=1.7689978769999999 podStartE2EDuration="1.768997877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:16.76724218 +0000 UTC m=+3798.338826090" watchObservedRunningTime="2026-11-03 12:54:16.768997877 +0000 UTC m=+3798.340581826"
Nov 03 12:54:27.672886 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:27.672852    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 12:54:28.179827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.179785    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"]
Nov 03 12:54:28.184583 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.184561    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.236432 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.236401    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"]
Nov 03 12:54:28.278175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.278144    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:54:28.278175 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.278175    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:54:28.278420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.278203    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:54:28.278420 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.278236    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:54:28.314827 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.314793    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.315006 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.314983    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.315103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.315084    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7tw\" (UniqueName: \"kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.416241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.416205    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.416462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.416277    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7tw\" (UniqueName: \"kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.416462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.416384    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.417000 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.416979    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.418830 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.418807    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.469118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.469090    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7tw\" (UniqueName: \"kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw\") pod \"maas-api-revoke-sub-4330eab5-cljxb\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.498440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.498405    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:28.887608 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:28.887573    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"]
Nov 03 12:54:28.888999 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:54:28.888967    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e06f2fe_3430_4ec5_a322_827609709d4b.slice/crio-1d3320b87a923613dbe525e0e67979f12bec051d417491d522bbb847b14cb28d WatchSource:0}: Error finding container 1d3320b87a923613dbe525e0e67979f12bec051d417491d522bbb847b14cb28d: Status 404 returned error can't find the container with id 1d3320b87a923613dbe525e0e67979f12bec051d417491d522bbb847b14cb28d
Nov 03 12:54:29.756620 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:29.756579    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb" event={"ID":"0e06f2fe-3430-4ec5-a322-827609709d4b","Type":"ContainerStarted","Data":"7e3b4cca12fd9a203433fc53ce92732f39e06136ec54726a4fabc2a6413d12ba"}
Nov 03 12:54:29.756864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:29.756647    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb" event={"ID":"0e06f2fe-3430-4ec5-a322-827609709d4b","Type":"ContainerStarted","Data":"1d3320b87a923613dbe525e0e67979f12bec051d417491d522bbb847b14cb28d"}
Nov 03 12:54:29.863413 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:29.863366    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb" podStartSLOduration=1.863351475 podStartE2EDuration="1.863351475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:29.860931709 +0000 UTC m=+3811.432515612" watchObservedRunningTime="2026-11-03 12:54:29.863351475 +0000 UTC m=+3811.434935386"
Nov 03 12:54:30.763403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:30.763371    4546 generic.go:356] "Generic (PLEG): container finished" podID="0e06f2fe-3430-4ec5-a322-827609709d4b" containerID="7e3b4cca12fd9a203433fc53ce92732f39e06136ec54726a4fabc2a6413d12ba" exitCode=0
Nov 03 12:54:30.763883 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:30.763455    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb" event={"ID":"0e06f2fe-3430-4ec5-a322-827609709d4b","Type":"ContainerDied","Data":"7e3b4cca12fd9a203433fc53ce92732f39e06136ec54726a4fabc2a6413d12ba"}
Nov 03 12:54:31.929894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:31.929865    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:32.061645 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.061552    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca\") pod \"0e06f2fe-3430-4ec5-a322-827609709d4b\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") "
Nov 03 12:54:32.061852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.061706    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca\") pod \"0e06f2fe-3430-4ec5-a322-827609709d4b\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") "
Nov 03 12:54:32.061852 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.061730    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw\" (UniqueName: \"kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw\") pod \"0e06f2fe-3430-4ec5-a322-827609709d4b\" (UID: \"0e06f2fe-3430-4ec5-a322-827609709d4b\") "
Nov 03 12:54:32.062064 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.062034    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca" pod "0e06f2fe-3430-4ec5-a322-827609709d4b" (UID: "0e06f2fe-3430-4ec5-a322-827609709d4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:54:32.062256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.062234    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:32.064042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.064013    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca" pod "0e06f2fe-3430-4ec5-a322-827609709d4b" (UID: "0e06f2fe-3430-4ec5-a322-827609709d4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:54:32.064200 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.064179    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw" pod "0e06f2fe-3430-4ec5-a322-827609709d4b" (UID: "0e06f2fe-3430-4ec5-a322-827609709d4b"). InnerVolumeSpecName "kube-api-access-mg7tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:54:32.163731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.163693    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e06f2fe-3430-4ec5-a322-827609709d4b-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:32.163731 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.163722    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7tw\" (UniqueName: \"kubernetes.io/projected/0e06f2fe-3430-4ec5-a322-827609709d4b-kube-api-access-mg7tw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:54:32.778254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.778224    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"
Nov 03 12:54:32.778439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.778253    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb" event={"ID":"0e06f2fe-3430-4ec5-a322-827609709d4b","Type":"ContainerDied","Data":"1d3320b87a923613dbe525e0e67979f12bec051d417491d522bbb847b14cb28d"}
Nov 03 12:54:32.778439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:32.778298    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d3320b87a923613dbe525e0e67979f12bec051d417491d522bbb847b14cb28d"
Nov 03 12:54:37.652433 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:54:37.652352    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:54:43.137895 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.137862    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"]
Nov 03 12:54:43.138458 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.138444    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0e06f2fe-3430-4ec5-a322-827609709d4b" containerName="revoke-keys"
Nov 03 12:54:43.138534 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.138459    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e06f2fe-3430-4ec5-a322-827609709d4b" containerName="revoke-keys"
Nov 03 12:54:43.138588 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.138551    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e06f2fe-3430-4ec5-a322-827609709d4b" containerName="revoke-keys"
Nov 03 12:54:43.142113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.142095    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.206684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.206644    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.207045 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.207023    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.215124 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.215095    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.215391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.215374    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.223773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.215542    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.224147 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.224116    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.247810 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.224373    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjml\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.250855 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.248203    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.251422 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.214962    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d126a6-openshift-default-dockercfg-bqkw2\""
Nov 03 12:54:43.253610 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.253588    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"]
Nov 03 12:54:43.253954 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.251370    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355275 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355249    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355303    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjml\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355373    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355394    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355472    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355497    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355538    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355568    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355594    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.355740 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355679    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.356040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355796    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.356040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355944    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.356040 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.355995    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.356256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.356231    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.358254 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.358232    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.358379 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.358360    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.411276 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.411210    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.411386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.411291    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjml\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml\") pod \"e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") " pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.455416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.455394    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:43.880257 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.880228    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"]
Nov 03 12:54:43.881096 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:54:43.881065    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932c34dc_3919_4ec7_bce3_50d81bdd1ddb.slice/crio-eec256e6c4a63267c3d85cdf45609beca8a7ea9c3559e141906a08058a517b80 WatchSource:0}: Error finding container eec256e6c4a63267c3d85cdf45609beca8a7ea9c3559e141906a08058a517b80: Status 404 returned error can't find the container with id eec256e6c4a63267c3d85cdf45609beca8a7ea9c3559e141906a08058a517b80
Nov 03 12:54:43.887198 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.887162    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:54:43.887314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.887248    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:54:43.887314 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:43.887295    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:54:44.869113 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:44.869075    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" event={"ID":"932c34dc-3919-4ec7-bce3-50d81bdd1ddb","Type":"ContainerStarted","Data":"0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"}
Nov 03 12:54:44.869541 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:44.869136    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" event={"ID":"932c34dc-3919-4ec7-bce3-50d81bdd1ddb","Type":"ContainerStarted","Data":"eec256e6c4a63267c3d85cdf45609beca8a7ea9c3559e141906a08058a517b80"}
Nov 03 12:54:45.024459 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:45.024396    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podStartSLOduration=2.024376479 podStartE2EDuration="2.024376479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:45.021993142 +0000 UTC m=+3826.593577058" watchObservedRunningTime="2026-11-03 12:54:45.024376479 +0000 UTC m=+3826.595960393"
Nov 03 12:54:45.455834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:45.455798    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:45.461401 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:45.461378    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:45.875659 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:45.875631    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:45.876729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:45.876710    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:54:51.308799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.305320    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"]
Nov 03 12:54:51.311213 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.311147    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.364149 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.364086    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"]
Nov 03 12:54:51.405815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.405783    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d126a6\""
Nov 03 12:54:51.405991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.405901    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d126a6-dockercfg-4qr6n\""
Nov 03 12:54:51.485111 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.485068    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.485298 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.485179    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mt86\" (UniqueName: \"kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86\") pod \"payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.586663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.586571    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.586663 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.586617    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mt86\" (UniqueName: \"kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86\") pod \"payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.587350 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.587328    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.639825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.639793    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mt86\" (UniqueName: \"kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86\") pod \"payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:51.931247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:51.931163    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:52.298416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:52.298383    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"]
Nov 03 12:54:52.298894 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:54:52.298864    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bf56ec_20f3_45c0_ade4_80192e2c7c21.slice/crio-14b99b96dae4e6f88197de398c53eddaf5125053653d8612e89e23be22deebf2 WatchSource:0}: Error finding container 14b99b96dae4e6f88197de398c53eddaf5125053653d8612e89e23be22deebf2: Status 404 returned error can't find the container with id 14b99b96dae4e6f88197de398c53eddaf5125053653d8612e89e23be22deebf2
Nov 03 12:54:52.927494 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:52.927453    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh" event={"ID":"c9bf56ec-20f3-45c0-ade4-80192e2c7c21","Type":"ContainerStarted","Data":"88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9"}
Nov 03 12:54:52.927963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:52.927528    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:54:52.927963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:52.927553    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh" event={"ID":"c9bf56ec-20f3-45c0-ade4-80192e2c7c21","Type":"ContainerStarted","Data":"14b99b96dae4e6f88197de398c53eddaf5125053653d8612e89e23be22deebf2"}
Nov 03 12:54:53.035441 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:53.035382    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh" podStartSLOduration=2.035364594 podStartE2EDuration="2.035364594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:53.031787218 +0000 UTC m=+3834.603371144" watchObservedRunningTime="2026-11-03 12:54:53.035364594 +0000 UTC m=+3834.606948508"
Nov 03 12:54:55.558293 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.558259    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"]
Nov 03 12:54:55.562765 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.562730    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.636839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.636787    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2slb\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.636881    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.636916    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637022 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.637001    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.637157    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637256 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.637245    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637297 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.637285    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637336 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.637314    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.637378 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.637367    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738375 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738342    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738408    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738430    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738453    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738485    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738547 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738532    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2slb\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738558    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738591    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738643    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738816    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.738879 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738826    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.739077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.738915    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.739077 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.739007    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.739368 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.739349    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.741012 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.740991    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.741287 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.741265    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.758722 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.758693    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"]
Nov 03 12:54:55.791643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.791618    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.791854 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.791838    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2slb\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7m2kj\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:55.875991 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:55.875910    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:56.291936 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:56.291906    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"]
Nov 03 12:54:56.328903 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:54:56.328863    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91662547_69e9_4583_b8d1_5ca79ff8e5c3.slice/crio-77a1943f855a5e97f40dc67a8aaa7bf675d6f5a329bd1c5a2be2825200444102 WatchSource:0}: Error finding container 77a1943f855a5e97f40dc67a8aaa7bf675d6f5a329bd1c5a2be2825200444102: Status 404 returned error can't find the container with id 77a1943f855a5e97f40dc67a8aaa7bf675d6f5a329bd1c5a2be2825200444102
Nov 03 12:54:56.335871 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:56.335839    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:54:56.335963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:56.335911    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:54:56.335963 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:56.335948    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:54:56.960299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:56.960260    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" event={"ID":"91662547-69e9-4583-b8d1-5ca79ff8e5c3","Type":"ContainerStarted","Data":"86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e"}
Nov 03 12:54:56.960711 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:56.960320    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" event={"ID":"91662547-69e9-4583-b8d1-5ca79ff8e5c3","Type":"ContainerStarted","Data":"77a1943f855a5e97f40dc67a8aaa7bf675d6f5a329bd1c5a2be2825200444102"}
Nov 03 12:54:57.134676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:57.134600    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podStartSLOduration=2.134578103 podStartE2EDuration="2.134578103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:57.127243448 +0000 UTC m=+3838.698827372" watchObservedRunningTime="2026-11-03 12:54:57.134578103 +0000 UTC m=+3838.706162016"
Nov 03 12:54:57.876095 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:57.876059    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:54:57.877640 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:57.877615    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:54:57.877770 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:57.877660    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:54:58.877069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:58.877031    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:54:58.877479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:58.877106    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:54:59.877020 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:59.876987    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:54:59.877229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:54:59.877042    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:55:00.880439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:00.879835    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:55:00.880439 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:00.879893    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:55:01.876403 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:01.876363    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:55:01.876634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:01.876429    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:55:02.876466 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:02.876423    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:55:02.876980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:02.876499    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:55:03.876510 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:03.876469    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Nov 03 12:55:03.877013 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:03.876545    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Nov 03 12:55:03.936948 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:03.936912    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 12:55:05.876980 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:05.876938    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:05.877447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:05.877021    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:06.917908 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:06.917874    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:06.918442 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:06.917924    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:07.407525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:07.407484    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:55:07.407874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:07.407853    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:55:07.408769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:07.408730    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 12:55:07.653988 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:55:07.653954    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:55:10.671849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.671816    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"]
Nov 03 12:55:10.675950 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.675931    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.686473 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686452    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.686595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686486    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.686658 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686613    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.686703 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686688    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.686769 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686739    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.686926 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686901    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rh9h\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.687009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.686953    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.687066 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.687023    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.687144 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.687127    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.775520 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.775483    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"]
Nov 03 12:55:10.788161 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788119    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788193    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788232    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788295    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rh9h\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788328    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788537    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788650    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788693 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788668    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788692    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788697    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.788856 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788722    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.789042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.788985    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.789042 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.789024    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.789495 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.789421    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.791282 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.791257    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.791383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.791324    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.841174 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.841136    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.841326 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.841232    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rh9h\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jzzd7\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:10.989646 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:10.989617    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:11.406328 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:11.406298    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"]
Nov 03 12:55:11.407309 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:55:11.407277    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75b32e5_fec4_4d87_b282_074e1ef9c8bb.slice/crio-6e242e3e6635885aa5ca30ece8a154f64107a6d8f0ff47693994bbe1b7da72fa WatchSource:0}: Error finding container 6e242e3e6635885aa5ca30ece8a154f64107a6d8f0ff47693994bbe1b7da72fa: Status 404 returned error can't find the container with id 6e242e3e6635885aa5ca30ece8a154f64107a6d8f0ff47693994bbe1b7da72fa
Nov 03 12:55:11.410054 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:11.410022    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:55:11.410164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:11.410099    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:55:11.410164 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:11.410145    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:55:12.088542 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:12.088501    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" event={"ID":"e75b32e5-fec4-4d87-b282-074e1ef9c8bb","Type":"ContainerStarted","Data":"39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122"}
Nov 03 12:55:12.089008 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:12.088589    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" event={"ID":"e75b32e5-fec4-4d87-b282-074e1ef9c8bb","Type":"ContainerStarted","Data":"6e242e3e6635885aa5ca30ece8a154f64107a6d8f0ff47693994bbe1b7da72fa"}
Nov 03 12:55:12.198795 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:12.198729    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podStartSLOduration=2.198715903 podStartE2EDuration="2.198715903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:55:12.196266801 +0000 UTC m=+3853.767850712" watchObservedRunningTime="2026-11-03 12:55:12.198715903 +0000 UTC m=+3853.770299814"
Nov 03 12:55:12.990589 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:12.990554    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:12.992347 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:12.992319    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:12.992486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:12.992380    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:13.990416 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:13.990382    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:13.990840 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:13.990443    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:14.990427 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:14.990393    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:14.990838 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:14.990446    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:15.990839 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:15.990806    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:15.991247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:15.990872    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:16.990016 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:16.989979    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:16.990215 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:16.990047    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:17.990511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:17.990469    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:17.990972 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:17.990541    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:18.990393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:18.990357    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jzzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 12:55:18.990632 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:18.990410    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 12:55:20.318515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:20.318479    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:20.319374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:20.319351    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:20.320246 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:20.320225    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 12:55:37.651533 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:55:37.651497    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:55:48.039418 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:48.039376    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"]
Nov 03 12:55:53.040629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.040591    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d" podUID="dddd777a-8170-415c-bc5e-47b844c113bb" containerName="payload-processing" containerID="cri-o://61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e" gracePeriod=25
Nov 03 12:55:53.307127 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.307072    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:55:53.393334 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.393308    4546 generic.go:356] "Generic (PLEG): container finished" podID="dddd777a-8170-415c-bc5e-47b844c113bb" containerID="61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e" exitCode=0
Nov 03 12:55:53.393488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.393348    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"
Nov 03 12:55:53.393488 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.393350    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d" event={"ID":"dddd777a-8170-415c-bc5e-47b844c113bb","Type":"ContainerDied","Data":"61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e"}
Nov 03 12:55:53.393569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.393489    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d" event={"ID":"dddd777a-8170-415c-bc5e-47b844c113bb","Type":"ContainerDied","Data":"5cc912cd8c45d8c807aa10c041e4361b9a287ffd1d903c4fb6a3cf2733758208"}
Nov 03 12:55:53.393569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.393518    4546 scope.go:122] "RemoveContainer" containerID="61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e"
Nov 03 12:55:53.402704 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.402690    4546 scope.go:122] "RemoveContainer" containerID="61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e"
Nov 03 12:55:53.402984 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:55:53.402964    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e\": container with ID starting with 61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e not found: ID does not exist" containerID="61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e"
Nov 03 12:55:53.403038 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.402990    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e"} err="failed to get container status \"61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e\": rpc error: code = NotFound desc = could not find container \"61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e\": container with ID starting with 61922a09bd897f0efa331ae57daa1421166e272b1c2442e412eda0cb0454323e not found: ID does not exist"
Nov 03 12:55:53.465901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.465872    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials\") pod \"dddd777a-8170-415c-bc5e-47b844c113bb\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") "
Nov 03 12:55:53.466063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.465936    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987\") pod \"dddd777a-8170-415c-bc5e-47b844c113bb\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") "
Nov 03 12:55:53.466155 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.466140    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume\") pod \"dddd777a-8170-415c-bc5e-47b844c113bb\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") "
Nov 03 12:55:53.466209 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.466196    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume\") pod \"dddd777a-8170-415c-bc5e-47b844c113bb\" (UID: \"dddd777a-8170-415c-bc5e-47b844c113bb\") "
Nov 03 12:55:53.466564 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.466526    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume" pod "dddd777a-8170-415c-bc5e-47b844c113bb" (UID: "dddd777a-8170-415c-bc5e-47b844c113bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:55:53.466676 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.466532    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume" pod "dddd777a-8170-415c-bc5e-47b844c113bb" (UID: "dddd777a-8170-415c-bc5e-47b844c113bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:55:53.466738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.466721    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:55:53.466815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.466742    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddd777a-8170-415c-bc5e-47b844c113bb-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:55:53.468182 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.468162    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987" pod "dddd777a-8170-415c-bc5e-47b844c113bb" (UID: "dddd777a-8170-415c-bc5e-47b844c113bb"). InnerVolumeSpecName "kube-api-access-qs987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:55:53.468241 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.468205    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials" pod "dddd777a-8170-415c-bc5e-47b844c113bb" (UID: "dddd777a-8170-415c-bc5e-47b844c113bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:55:53.568381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.568304    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:55:53.568381 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.568331    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs987\" (UniqueName: \"kubernetes.io/projected/dddd777a-8170-415c-bc5e-47b844c113bb-kube-api-access-qs987\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:55:53.807069 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.807035    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"]
Nov 03 12:55:53.811989 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:53.811967    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94e247/payload-processing-external-model-e2e-extmod-b-94e247-7959zmm9d"]
Nov 03 12:55:54.573252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:55:54.573225    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddd777a-8170-415c-bc5e-47b844c113bb" path="/var/lib/kubelet/pods/dddd777a-8170-415c-bc5e-47b844c113bb/volumes"
Nov 03 12:56:07.652366 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:56:07.652269    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:56:23.457776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:23.456961    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:56:23.457776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:23.457063    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:56:23.558776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:23.557849    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:56:31.650593 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.650557    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"]
Nov 03 12:56:31.651374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.651349    4546 cpu_manager.go:414] "containerMap: removing container" podUID="dddd777a-8170-415c-bc5e-47b844c113bb" containerName="payload-processing"
Nov 03 12:56:31.651374 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.651374    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddd777a-8170-415c-bc5e-47b844c113bb" containerName="payload-processing"
Nov 03 12:56:31.651568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.651513    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddd777a-8170-415c-bc5e-47b844c113bb" containerName="payload-processing"
Nov 03 12:56:31.655438 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.655418    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.707865 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.707831    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"]
Nov 03 12:56:31.751925 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.751890    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:56:31.752170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.751902    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:56:31.752170 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.751967    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 12:56:31.752398 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.752010    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 12:56:31.804302 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.804269    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.804552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.804525    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cs72\" (UniqueName: \"kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.804649 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.804635    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.905881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.905789    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cs72\" (UniqueName: \"kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.905881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.905870    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.906136 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.905938    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.906595 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.906569    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.908530 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.908503    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.959373 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.959336    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cs72\" (UniqueName: \"kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72\") pod \"maas-api-revoke-sub-f7fd9281-rkrg4\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:31.973639 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:31.973611    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:32.343580 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:32.343548    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"]
Nov 03 12:56:32.344360 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:56:32.344332    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8b4d25_8fb8_4c21_8b3f_35ef63a5918f.slice/crio-b98301977b74210ebbc5d2afcc964137989ddea6f30741972cefd58eb0b24b99 WatchSource:0}: Error finding container b98301977b74210ebbc5d2afcc964137989ddea6f30741972cefd58eb0b24b99: Status 404 returned error can't find the container with id b98301977b74210ebbc5d2afcc964137989ddea6f30741972cefd58eb0b24b99
Nov 03 12:56:32.702229 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:32.702197    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4" event={"ID":"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f","Type":"ContainerStarted","Data":"37ababcb573adf8284627224a6719ab666625f182164c91914e0973f05790928"}
Nov 03 12:56:32.702702 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:32.702257    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4" event={"ID":"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f","Type":"ContainerStarted","Data":"b98301977b74210ebbc5d2afcc964137989ddea6f30741972cefd58eb0b24b99"}
Nov 03 12:56:32.807269 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:32.807217    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4" podStartSLOduration=1.807204372 podStartE2EDuration="1.807204372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:32.806644749 +0000 UTC m=+3934.378228661" watchObservedRunningTime="2026-11-03 12:56:32.807204372 +0000 UTC m=+3934.378788283"
Nov 03 12:56:33.710060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:33.709971    4546 generic.go:356] "Generic (PLEG): container finished" podID="ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" containerID="37ababcb573adf8284627224a6719ab666625f182164c91914e0973f05790928" exitCode=0
Nov 03 12:56:33.710060 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:33.710047    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4" event={"ID":"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f","Type":"ContainerDied","Data":"37ababcb573adf8284627224a6719ab666625f182164c91914e0973f05790928"}
Nov 03 12:56:34.888482 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.888455    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:34.951239 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.951203    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca\") pod \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") "
Nov 03 12:56:34.951440 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.951385    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72\" (UniqueName: \"kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72\") pod \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") "
Nov 03 12:56:34.951512 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.951480    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca\") pod \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\" (UID: \"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f\") "
Nov 03 12:56:34.951572 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.951532    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca" pod "ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" (UID: "ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:56:34.951944 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.951925    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:34.953574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.953550    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca" pod "ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" (UID: "ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:56:34.953708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:34.953592    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72" pod "ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" (UID: "ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f"). InnerVolumeSpecName "kube-api-access-8cs72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:56:35.053041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:35.052954    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cs72\" (UniqueName: \"kubernetes.io/projected/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-kube-api-access-8cs72\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:35.053041 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:35.052984    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:56:35.725959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:35.725930    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"
Nov 03 12:56:35.726142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:35.725978    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4" event={"ID":"ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f","Type":"ContainerDied","Data":"b98301977b74210ebbc5d2afcc964137989ddea6f30741972cefd58eb0b24b99"}
Nov 03 12:56:35.726142 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:35.726025    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98301977b74210ebbc5d2afcc964137989ddea6f30741972cefd58eb0b24b99"
Nov 03 12:56:37.651483 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:56:37.651447    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:56:44.743299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.743259    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"]
Nov 03 12:56:44.744114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.744094    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" containerName="revoke-keys"
Nov 03 12:56:44.744114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.744115    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" containerName="revoke-keys"
Nov 03 12:56:44.744273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.744263    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" containerName="revoke-keys"
Nov 03 12:56:44.747957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.747938    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.849967 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.849933    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9ec01e4d84c9\""
Nov 03 12:56:44.850173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.849983    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 12:56:44.850338 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.850315    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 12:56:44.850447 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.850322    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 12:56:44.850525 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.850441    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 12:56:44.854025 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.854004    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"]
Nov 03 12:56:44.871087 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.871064    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.871252 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.871230    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq66d\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.871352 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.871339    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.871391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.871365    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.904063 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.904038    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6lkbc\""
Nov 03 12:56:44.972160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.972128    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.972160 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.972165    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.972412 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.972357    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.972552 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.972531    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq66d\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.973058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.973037    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.973179 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.973037    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:44.974904 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:44.974884    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:45.025623 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.025540    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq66d\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d\") pod \"payload-processing-external-model-7b96957bd6-zd8h9\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") " pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:45.062825 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.062791    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:45.533655 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.533631    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"]
Nov 03 12:56:45.534824 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:56:45.534792    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode539c7bc_493c_4b01_a7ad_4f3ab3dc4e43.slice/crio-99eca470794ff5d29d9257a94f4a4945a8acd9ce5c42bb8cbbc28370840ca78e WatchSource:0}: Error finding container 99eca470794ff5d29d9257a94f4a4945a8acd9ce5c42bb8cbbc28370840ca78e: Status 404 returned error can't find the container with id 99eca470794ff5d29d9257a94f4a4945a8acd9ce5c42bb8cbbc28370840ca78e
Nov 03 12:56:45.794791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.794676    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" event={"ID":"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43","Type":"ContainerStarted","Data":"c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28"}
Nov 03 12:56:45.794791 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.794738    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" event={"ID":"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43","Type":"ContainerStarted","Data":"99eca470794ff5d29d9257a94f4a4945a8acd9ce5c42bb8cbbc28370840ca78e"}
Nov 03 12:56:45.795222 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.794804    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 12:56:45.902662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:45.902610    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podStartSLOduration=1.902595456 podStartE2EDuration="1.902595456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:45.900481579 +0000 UTC m=+3947.472065511" watchObservedRunningTime="2026-11-03 12:56:45.902595456 +0000 UTC m=+3947.474179420"
Nov 03 12:56:55.628638 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.628604    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"]
Nov 03 12:56:55.632928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.632898    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.684462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.684425    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.684660 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.684577    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb55k\" (UniqueName: \"kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k\") pod \"payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.691738 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.691701    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"]
Nov 03 12:56:55.776629 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.732237    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-532442\""
Nov 03 12:56:55.778349 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.735267    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-532442-dockercfg-wgc24\""
Nov 03 12:56:55.811002 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.810942    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.811391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.811366    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb55k\" (UniqueName: \"kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k\") pod \"payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.813079 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.813038    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.867567 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.867533    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb55k\" (UniqueName: \"kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k\") pod \"payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:55.947630 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:55.947544    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:56.317523 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:56.317496    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"]
Nov 03 12:56:56.320214 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:56:56.320185    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d82fb07_7cb7_450c_ba71_7a533dc9603b.slice/crio-238c117a17c5a4f31717a200535eceb7f3211ae5b31d9c857574dee3402c665f WatchSource:0}: Error finding container 238c117a17c5a4f31717a200535eceb7f3211ae5b31d9c857574dee3402c665f: Status 404 returned error can't find the container with id 238c117a17c5a4f31717a200535eceb7f3211ae5b31d9c857574dee3402c665f
Nov 03 12:56:56.805337 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:56.805294    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:56:56.872633 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:56.872600    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8" event={"ID":"6d82fb07-7cb7-450c-ba71-7a533dc9603b","Type":"ContainerStarted","Data":"a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c"}
Nov 03 12:56:56.872860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:56.872651    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8" event={"ID":"6d82fb07-7cb7-450c-ba71-7a533dc9603b","Type":"ContainerStarted","Data":"238c117a17c5a4f31717a200535eceb7f3211ae5b31d9c857574dee3402c665f"}
Nov 03 12:56:56.872860 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:56.872696    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:56:56.979718 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:56.979660    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8" podStartSLOduration=1.9796456519999999 podStartE2EDuration="1.979645652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:56.977332863 +0000 UTC m=+3958.548916795" watchObservedRunningTime="2026-11-03 12:56:56.979645652 +0000 UTC m=+3958.551229603"
Nov 03 12:56:57.817477 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.817446    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"]
Nov 03 12:56:57.821849 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.821831    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851203 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851161    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851223    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851341 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851311    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851433 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851347    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851474 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851433    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851526 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851510    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851643 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851627    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlz2w\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851685 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851671    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.851808 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.851795    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.870335 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.870306    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fb74b5-openshift-default-dockercfg-dx98f\""
Nov 03 12:56:57.923026 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.922964    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"]
Nov 03 12:56:57.953377 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.953339    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.953562 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.953484    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.953662 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.953562    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.953962 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.953940    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954019    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954076 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954030    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954296 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954271    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954388 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954342    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954450 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954396    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954452    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954574 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954547    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954635 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954605    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlz2w\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.954864 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.954834    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.955082 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.955033    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.956909 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.956883    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:57.957596 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:57.957572    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:58.006959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.006922    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:58.007493 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.007468    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlz2w\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w\") pod \"e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") " pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:58.134720 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.134624    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:58.578669 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:56:58.578640    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc20d22_0d86_45f9_af45_6af554757811.slice/crio-8f07c5c4e636580632ed2deb768c8ad549ef33a141b92dbf613a097f7cd97412 WatchSource:0}: Error finding container 8f07c5c4e636580632ed2deb768c8ad549ef33a141b92dbf613a097f7cd97412: Status 404 returned error can't find the container with id 8f07c5c4e636580632ed2deb768c8ad549ef33a141b92dbf613a097f7cd97412
Nov 03 12:56:58.581356 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.581334    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"]
Nov 03 12:56:58.582031 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.581999    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:56:58.582122 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.582085    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:56:58.582195 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.582132    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:56:58.892460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.892367    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" event={"ID":"2cc20d22-0d86-45f9-af45-6af554757811","Type":"ContainerStarted","Data":"b5167c42a5967110250e8e6d867d0ca6ca757db51f436797068051db9d32de98"}
Nov 03 12:56:58.892460 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:58.892420    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" event={"ID":"2cc20d22-0d86-45f9-af45-6af554757811","Type":"ContainerStarted","Data":"8f07c5c4e636580632ed2deb768c8ad549ef33a141b92dbf613a097f7cd97412"}
Nov 03 12:56:59.005569 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:59.005513    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podStartSLOduration=2.005498242 podStartE2EDuration="2.005498242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:58.999991616 +0000 UTC m=+3960.571575539" watchObservedRunningTime="2026-11-03 12:56:59.005498242 +0000 UTC m=+3960.577082214"
Nov 03 12:56:59.135957 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:59.135099    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:59.142103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:59.142072    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:59.899708 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:59.899673    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:56:59.900862 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:56:59.900835    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 12:57:01.761273 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:01.761240    4546 scope.go:122] "RemoveContainer" containerID="b2fb29d4b204d5a11992145403917dd3a3e98f057a426d077f12cc71f2731aa0"
Nov 03 12:57:06.804455 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:06.804399    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:57:07.695473 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:57:07.695431    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:57:07.931780 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:07.927955    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 12:57:08.805971 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:08.805916    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.040894 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.040839    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.210832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.210556    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.279968 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.279901    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.458247 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.458193    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.559777 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.559009    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 12:57:09.597775 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.596840    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 12:57:09.620906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.620864    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.658901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.658850    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.744906 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.744861    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:09.812831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.811625    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 12:57:09.812831 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.812079    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:57:09.956776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:09.953071    4546 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:10.045882 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.045839    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:10.062786 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.051532    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:57:10.369861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.369826    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:57:10.370386 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.370369    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:57:10.442778 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.429363    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 12:57:10.508299 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.508269    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt"
Nov 03 12:57:10.564776 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.560823    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:10.581782 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.565688    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:57:10.643832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.627877    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:10.705047 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.705015    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:57:10.706732 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.705940    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 12:57:10.711030 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.711013    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 12:57:10.716901 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:10.716877    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 12:57:15.387407 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.387376    4546 generic.go:356] "Generic (PLEG): container finished" podID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerID="0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2" exitCode=0
Nov 03 12:57:15.387890 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.387451    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" event={"ID":"932c34dc-3919-4ec7-bce3-50d81bdd1ddb","Type":"ContainerDied","Data":"0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"}
Nov 03 12:57:15.387959 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.387942    4546 scope.go:122] "RemoveContainer" containerID="0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"
Nov 03 12:57:15.389737 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.389706    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:57:15.389874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.389805    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:57:15.389874 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.389842    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:57:15.875832 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:15.875801    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:57:16.395342 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:16.395309    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" event={"ID":"932c34dc-3919-4ec7-bce3-50d81bdd1ddb","Type":"ContainerStarted","Data":"292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618"}
Nov 03 12:57:16.456516 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:16.456485    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:57:16.457868 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:16.457848    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 12:57:16.457945 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:16.457891    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 12:57:16.803721 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:16.803680    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:57:17.455824 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:17.455793    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 12:57:17.456309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:17.455848    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 12:57:18.456678 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:18.456635    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 12:57:18.457114 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:18.456695    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 12:57:19.456479 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:19.456441    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 12:57:19.456696 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:19.456508    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 12:57:20.456383 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:20.456349    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 12:57:20.456818 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:20.456404    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 12:57:21.456369 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:21.456330    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 12:57:21.456621 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:21.456384    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 12:57:22.460506 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:22.460477    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:57:22.460993 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:22.460980    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:57:22.461634 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:22.461610    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 12:57:26.803650 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:26.803610    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:57:36.803798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:36.803742    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:57:37.235905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.235876    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"]
Nov 03 12:57:37.240268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.240250    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.290173 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.290138    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-aa9221-openshift-default-dockercfg-czjlz\""
Nov 03 12:57:37.401118 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401079    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401145    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401179    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401211    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401242    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401344 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401328    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tqpr\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401452    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401476    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.401699 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.401527    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.456793 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.456728    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"]
Nov 03 12:57:37.503568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.503568 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503523    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.503880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503570    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.503880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503602    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.503880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503647    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.503880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503737    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tqpr\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.503880 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503876    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.504214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503902    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.504214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503964    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.504214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.503975    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.504214 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.504040    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.504457 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.504309    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.504689 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.504668    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.505125 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.505091    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.506888 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.506865    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.508225 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.508199    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.556905 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.556872    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.557058 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.557028    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tqpr\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr\") pod \"e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") " pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:37.651545 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:57:37.651469    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:57:37.853834 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:37.853719    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:38.269453 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.269421    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"]
Nov 03 12:57:38.269838 ip-10-0-136-245 kubenswrapper[4546]: W1103 12:57:38.269812    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded666c5a_f22a_44f2_842c_b350cf68825e.slice/crio-412605148ff2be6bd79e8208da7301aec3d5c41b17908bd09eae1b33c901beeb WatchSource:0}: Error finding container 412605148ff2be6bd79e8208da7301aec3d5c41b17908bd09eae1b33c901beeb: Status 404 returned error can't find the container with id 412605148ff2be6bd79e8208da7301aec3d5c41b17908bd09eae1b33c901beeb
Nov 03 12:57:38.272428 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.272398    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:57:38.272511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.272460    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:57:38.272511 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.272488    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:57:38.573911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.573830    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" event={"ID":"ed666c5a-f22a-44f2-842c-b350cf68825e","Type":"ContainerStarted","Data":"52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"}
Nov 03 12:57:38.573911 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.573888    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" event={"ID":"ed666c5a-f22a-44f2-842c-b350cf68825e","Type":"ContainerStarted","Data":"412605148ff2be6bd79e8208da7301aec3d5c41b17908bd09eae1b33c901beeb"}
Nov 03 12:57:38.681798 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.681727    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podStartSLOduration=1.6817096089999999 podStartE2EDuration="1.681709609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:57:38.678943262 +0000 UTC m=+4000.250527193" watchObservedRunningTime="2026-11-03 12:57:38.681709609 +0000 UTC m=+4000.253293523"
Nov 03 12:57:38.854714 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:38.854636    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:39.860773 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:39.860718    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:40.579881 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:40.579845    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:40.580861 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:40.580842    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:57:46.803517 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:46.803478    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:57:49.567603 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:49.567564    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:57:55.043684 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:55.043652    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"]
Nov 03 12:57:55.089508 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:55.089479    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f495b88-q98j5"]
Nov 03 12:57:56.574312 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:56.574276    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d808ede0-6792-4ba9-90f9-8026162e1404" path="/var/lib/kubelet/pods/d808ede0-6792-4ba9-90f9-8026162e1404/volumes"
Nov 03 12:57:59.567729 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:57:59.567692    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:58:06.042187 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:06.042152    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"]
Nov 03 12:58:06.089551 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:06.089523    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3446e3d6-7rqpt"]
Nov 03 12:58:06.573892 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:06.573865    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a1bebf-5b9b-4940-99a3-e03edc4aa3b1" path="/var/lib/kubelet/pods/07a1bebf-5b9b-4940-99a3-e03edc4aa3b1/volumes"
Nov 03 12:58:07.674416 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:58:07.674377    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:58:09.568320 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:09.568277    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:58:19.568003 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:19.567964    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:58:29.567651 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:29.567615    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:58:37.651512 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:58:37.651475    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:58:39.567577 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:39.567535    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:58:49.568309 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:49.568269    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:58:59.567515 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:58:59.567476    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:59:01.797724 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:01.797686    4546 scope.go:122] "RemoveContainer" containerID="7b58b64b5e4d0298155a8189cc2a47bc780406f3a506e089d771b966984557ec"
Nov 03 12:59:01.807910 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:01.807888    4546 scope.go:122] "RemoveContainer" containerID="0b7adbf56f5a3a777cbba4f16cbd35d806f304e3da31cd2c28288be9db1cab04"
Nov 03 12:59:07.652369 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:59:07.652289    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:59:09.567464 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:09.567425    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:59:17.567394 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:17.567354    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:59:27.567255 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:27.567220    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:59:32.045103 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:32.045069    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"]
Nov 03 12:59:32.090129 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:32.090098    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4330eab5-cljxb"]
Nov 03 12:59:32.574408 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:32.574371    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e06f2fe-3430-4ec5-a322-827609709d4b" path="/var/lib/kubelet/pods/0e06f2fe-3430-4ec5-a322-827609709d4b/volumes"
Nov 03 12:59:37.567406 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:37.567375    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:59:37.651516 ip-10-0-136-245 kubenswrapper[4546]: E1103 12:59:37.651485    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 12:59:38.443221 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:38.443187    4546 generic.go:356] "Generic (PLEG): container finished" podID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerID="52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2" exitCode=0
Nov 03 12:59:38.443393 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:38.443263    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" event={"ID":"ed666c5a-f22a-44f2-842c-b350cf68825e","Type":"ContainerDied","Data":"52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"}
Nov 03 12:59:38.443668 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:38.443648    4546 scope.go:122] "RemoveContainer" containerID="52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"
Nov 03 12:59:38.445289 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:38.445264    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:59:38.445354 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:38.445339    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:59:38.445391 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:38.445369    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 12:59:39.452131 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:39.452098    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" event={"ID":"ed666c5a-f22a-44f2-842c-b350cf68825e","Type":"ContainerStarted","Data":"1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd"}
Nov 03 12:59:39.854015 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:39.853982    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:59:39.855462 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:39.855436    4546 patch_prober.go:29] interesting pod/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 12:59:39.855615 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:39.855483    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 12:59:40.581417 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:40.581386    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:59:40.854372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:40.854286    4546 patch_prober.go:29] interesting pod/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 12:59:40.854372 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:40.854346    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 12:59:41.854995 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:41.854963    4546 patch_prober.go:29] interesting pod/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 12:59:41.855414 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:41.855022    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 12:59:42.854799 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:42.854728    4546 patch_prober.go:29] interesting pod/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 12:59:42.855009 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:42.854824    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 12:59:43.854380 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:43.854345    4546 patch_prober.go:29] interesting pod/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 12:59:43.854815 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:43.854399    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 12:59:44.854486 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:44.854444    4546 patch_prober.go:29] interesting pod/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 12:59:44.854999 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:44.854527    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 12:59:45.858845 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:45.858812    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:59:45.859928 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:45.859908    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 12:59:47.567268 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:47.567235    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 12:59:57.567212 ip-10-0-136-245 kubenswrapper[4546]: I1103 12:59:57.567177    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:00.163535 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.163491    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"]
Nov 03 13:00:00.169023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.168991    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"]
Nov 03 13:00:00.169218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.169137    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.174124 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.174102    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"]
Nov 03 13:00:00.174258 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.174215    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.179072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.179050    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.210259 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.210230    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"]
Nov 03 13:00:00.214537 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.214515    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"]
Nov 03 13:00:00.214695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.214644    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.219339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.219318    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"]
Nov 03 13:00:00.219468 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.219417    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.223981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.223962    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"]
Nov 03 13:00:00.224136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.224117    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.228027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.227999    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"]
Nov 03 13:00:00.228134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.228117    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.232000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.231983    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"]
Nov 03 13:00:00.232110 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.232095    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.235890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.235866    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"]
Nov 03 13:00:00.235996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.235969    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.239851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.239831    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.252619 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.252599    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:00:00.252779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.252723    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:00:00.252856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.252791    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:00:00.256073 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256048    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tjn\" (UniqueName: \"kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.256196 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256164    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.256408 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256385    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7b7\" (UniqueName: \"kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.256932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256449    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.256932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256458    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"]
Nov 03 13:00:00.256932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256541    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkwh\" (UniqueName: \"kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.256932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.256642    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.261242 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.261214    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.262277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.262260    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"]
Nov 03 13:00:00.266363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.266344    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"]
Nov 03 13:00:00.266477 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.266444    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.270844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.270822    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"]
Nov 03 13:00:00.270937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.270909    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.271779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.271731    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"]
Nov 03 13:00:00.274607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.274589    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"]
Nov 03 13:00:00.277638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.277618    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"]
Nov 03 13:00:00.314065 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.314024    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"]
Nov 03 13:00:00.332952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.332918    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"]
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.358630    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmknq\" (UniqueName: \"kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.358684    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vns4c\" (UniqueName: \"kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.358727    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359055    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkwh\" (UniqueName: \"kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359097    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrx9\" (UniqueName: \"kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359143    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.359162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359172    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4zb\" (UniqueName: \"kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.359710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359205    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9n5\" (UniqueName: \"kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.359710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359239    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dh48\" (UniqueName: \"kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48\") pod \"maas-api-key-cleanup-29895180-v25ql\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.359710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359275    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cr5\" (UniqueName: \"kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.359710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359328    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknx7\" (UniqueName: \"kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7\") pod \"maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.359710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359370    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tjn\" (UniqueName: \"kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.359427    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361182    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361242    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895180-v25ql\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361282    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361348    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361399    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361422    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361460    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361522    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jpt\" (UniqueName: \"kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361549    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361581    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7b7\" (UniqueName: \"kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361609    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvb25\" (UniqueName: \"kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25\") pod \"maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361663    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.361711    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.363045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.362272    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.368870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.363877    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.368870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.363928    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"]
Nov 03 13:00:00.368870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.364611    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.373907 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.373882    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"]
Nov 03 13:00:00.379809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.379783    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"]
Nov 03 13:00:00.384819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.384793    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"]
Nov 03 13:00:00.400592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.400560    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"]
Nov 03 13:00:00.400895 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.400876    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"]
Nov 03 13:00:00.416045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.415975    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tjn\" (UniqueName: \"kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.416936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.416866    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkwh\" (UniqueName: \"kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.419172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.419153    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7b7\" (UniqueName: \"kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.419853 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.419827    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"]
Nov 03 13:00:00.462564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462535    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.462760 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462585    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895180-v25ql\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.462760 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462618    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.462760 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462656    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.462760 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462693    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.463074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462837    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.463074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462915    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.463074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.462999    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jpt\" (UniqueName: \"kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.463074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463041    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.463294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463080    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvb25\" (UniqueName: \"kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25\") pod \"maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.463294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463168    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.463294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463217    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmknq\" (UniqueName: \"kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.463294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463247    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vns4c\" (UniqueName: \"kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.463294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463290    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrx9\" (UniqueName: \"kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463380    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463432    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463458    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895180-v25ql\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463458    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463385    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4zb\" (UniqueName: \"kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.463560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463531    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9n5\" (UniqueName: \"kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.463977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463571    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dh48\" (UniqueName: \"kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48\") pod \"maas-api-key-cleanup-29895180-v25ql\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.463977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463687    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cr5\" (UniqueName: \"kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.463977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.463786    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknx7\" (UniqueName: \"kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7\") pod \"maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.464229 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.464203    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.464298 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.464247    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.464365 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.464329    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.464427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.464329    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.464581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.464560    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.464720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.464697    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.487159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.487134    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:00.494894 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.494876    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:00.500828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.500810    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:00.519504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.519197    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jpt\" (UniqueName: \"kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.522695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.522254    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvb25\" (UniqueName: \"kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25\") pod \"maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.525059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.524988    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dh48\" (UniqueName: \"kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48\") pod \"maas-api-key-cleanup-29895180-v25ql\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.527570 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.527489    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4zb\" (UniqueName: \"kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.528823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.528496    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vns4c\" (UniqueName: \"kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.529723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.529179    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9n5\" (UniqueName: \"kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5\") pod \"maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.529723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.529675    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknx7\" (UniqueName: \"kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7\") pod \"maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.530207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.529834    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmknq\" (UniqueName: \"kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.530410 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.530357    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrx9\" (UniqueName: \"kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.531161 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.530937    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:00.532847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.532808    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cr5\" (UniqueName: \"kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5\") pod \"maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.539388 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.539306    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:00.549271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.548990    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:00.558977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.558900    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:00.564185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.563996    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:00.576380 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.574985    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:00.598163 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.593788    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:00.604550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.604346    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:00.629720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.629689    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:00.631055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.631030    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:00.871430 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:00.871388    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"]
Nov 03 13:00:01.074965 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:01.074933    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201264ad_f94b_412c_ba91_24f933679a7e.slice/crio-2efeb9fc6d5ba7fa0c61fe8368c99ac795766028fc5f8a563bb53db710c3bc6a WatchSource:0}: Error finding container 2efeb9fc6d5ba7fa0c61fe8368c99ac795766028fc5f8a563bb53db710c3bc6a: Status 404 returned error can't find the container with id 2efeb9fc6d5ba7fa0c61fe8368c99ac795766028fc5f8a563bb53db710c3bc6a
Nov 03 13:00:01.075477 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.075419    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"]
Nov 03 13:00:01.122537 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.122511    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"]
Nov 03 13:00:01.129047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.129018    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"]
Nov 03 13:00:01.149848 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:01.149817    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aaace4b_7c89_47ad_be3e_9582d747f3dd.slice/crio-3dcd1f191a8d79d14eaea5287227b2912efd0fd1fac3f9eda82c871f69339de3 WatchSource:0}: Error finding container 3dcd1f191a8d79d14eaea5287227b2912efd0fd1fac3f9eda82c871f69339de3: Status 404 returned error can't find the container with id 3dcd1f191a8d79d14eaea5287227b2912efd0fd1fac3f9eda82c871f69339de3
Nov 03 13:00:01.173924 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.173896    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"]
Nov 03 13:00:01.179970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.179947    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"]
Nov 03 13:00:01.184479 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:01.184345    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f55a1dd_f311_44ad_9c0c_fcee328c98a3.slice/crio-457b838780082092f95703b682f1affca144a13ee20d0792f6d04dca7bde7ee4 WatchSource:0}: Error finding container 457b838780082092f95703b682f1affca144a13ee20d0792f6d04dca7bde7ee4: Status 404 returned error can't find the container with id 457b838780082092f95703b682f1affca144a13ee20d0792f6d04dca7bde7ee4
Nov 03 13:00:01.190154 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:01.190133    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1108ffa1_92ac_43f3_95c5_43d0fae9beed.slice/crio-0ed21a4ee61a1787cd97c0521c203050a55ff7e0d63100714b2df5832d6c5939 WatchSource:0}: Error finding container 0ed21a4ee61a1787cd97c0521c203050a55ff7e0d63100714b2df5832d6c5939: Status 404 returned error can't find the container with id 0ed21a4ee61a1787cd97c0521c203050a55ff7e0d63100714b2df5832d6c5939
Nov 03 13:00:01.252945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.252907    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"]
Nov 03 13:00:01.265994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.265970    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"]
Nov 03 13:00:01.293511 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.293476    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"]
Nov 03 13:00:01.349054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.349017    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"]
Nov 03 13:00:01.352602 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.352577    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"]
Nov 03 13:00:01.364692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.364663    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"]
Nov 03 13:00:01.364816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.364730    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"]
Nov 03 13:00:01.388218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.388186    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"]
Nov 03 13:00:01.420888 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:01.419681    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e1e24d_d7ef_4b81_a640_ed4cd8733dcf.slice/crio-6ea34f2843f859bab58040f714c262250f1cb65eec737c25f479f054093019d9 WatchSource:0}: Error finding container 6ea34f2843f859bab58040f714c262250f1cb65eec737c25f479f054093019d9: Status 404 returned error can't find the container with id 6ea34f2843f859bab58040f714c262250f1cb65eec737c25f479f054093019d9
Nov 03 13:00:01.688880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.688724    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerStarted","Data":"8718493aefd29c076b6e449a59444aef7517b0b6bc561f9450c46d217e86127c"}
Nov 03 13:00:01.690314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.690288    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerStarted","Data":"457b838780082092f95703b682f1affca144a13ee20d0792f6d04dca7bde7ee4"}
Nov 03 13:00:01.691737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.691710    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerStarted","Data":"f57c97716f70f45fd30ca7345e37e977ddd7aa02ea553265a53f3c7aaecdf0a7"}
Nov 03 13:00:01.692932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.692901    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" event={"ID":"2d847af6-74a2-45cb-9a6b-74750275610e","Type":"ContainerStarted","Data":"7b4f73f08d8907d2a310a8a121f1ccfa1862a83c9e2887a65d46a4c3facf80e4"}
Nov 03 13:00:01.694105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.694079    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerStarted","Data":"06008f71eb58c46300dbc795222dfd0e8f9163f2c054efe0961326bef4c02261"}
Nov 03 13:00:01.695424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.695402    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerStarted","Data":"4b87808511e827c541618f281e0a796fd6c8cc7122e64bf4eaa375eccebc5f72"}
Nov 03 13:00:01.696884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.696861    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerStarted","Data":"6d014b66c3a0b6b86732943e5d0686baec35cbd1eefdb1fdccfda03f6ecb93bd"}
Nov 03 13:00:01.696985 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.696906    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerStarted","Data":"2efeb9fc6d5ba7fa0c61fe8368c99ac795766028fc5f8a563bb53db710c3bc6a"}
Nov 03 13:00:01.698124 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.698104    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerStarted","Data":"0ed21a4ee61a1787cd97c0521c203050a55ff7e0d63100714b2df5832d6c5939"}
Nov 03 13:00:01.699413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.699391    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerStarted","Data":"a05a936ea073f6a154e718a4be5e9594caf7aa9118fc93ea94fb9c22a3e4af45"}
Nov 03 13:00:01.701349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.701324    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" event={"ID":"4dc7faed-4d52-4e8d-aed3-149fd75522a6","Type":"ContainerStarted","Data":"af62bd23aabed028cb0a982fcd234fa92712cd5042cf6f59cf647e1e6cb11201"}
Nov 03 13:00:01.702710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.702682    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerStarted","Data":"6ea34f2843f859bab58040f714c262250f1cb65eec737c25f479f054093019d9"}
Nov 03 13:00:01.704236 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.704211    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerStarted","Data":"3dcd1f191a8d79d14eaea5287227b2912efd0fd1fac3f9eda82c871f69339de3"}
Nov 03 13:00:01.705601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.705581    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" event={"ID":"bc292f12-2b0a-4b5a-ac16-416e64c9040c","Type":"ContainerStarted","Data":"a32e0dea084a69928b04622fd1e1511427057fa574b84b07c691f8a3fe918437"}
Nov 03 13:00:01.805905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.805853    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" podStartSLOduration=1.805837231 podStartE2EDuration="1.805837231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:01.803261386 +0000 UTC m=+4143.374845298" watchObservedRunningTime="2026-11-03 13:00:01.805837231 +0000 UTC m=+4143.377421197"
Nov 03 13:00:01.836420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:01.836398    4546 scope.go:122] "RemoveContainer" containerID="9701b77492324e519c36d8b5f07cd22fa3d0c90e1b24c41e8b1836fb01227037"
Nov 03 13:00:02.724906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.724870    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerStarted","Data":"617d18e571e1bee0d2d403b8d2f8d04479fa9c795b8f12f13b44e111566fff96"}
Nov 03 13:00:02.728620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.728592    4546 generic.go:356] "Generic (PLEG): container finished" podID="201264ad-f94b-412c-ba91-24f933679a7e" containerID="6d014b66c3a0b6b86732943e5d0686baec35cbd1eefdb1fdccfda03f6ecb93bd" exitCode=60
Nov 03 13:00:02.728813 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.728671    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerDied","Data":"6d014b66c3a0b6b86732943e5d0686baec35cbd1eefdb1fdccfda03f6ecb93bd"}
Nov 03 13:00:02.729209 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.729083    4546 scope.go:122] "RemoveContainer" containerID="6d014b66c3a0b6b86732943e5d0686baec35cbd1eefdb1fdccfda03f6ecb93bd"
Nov 03 13:00:02.730691 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.730634    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerStarted","Data":"b8884dba6e9df0fb95323b60a56f1eaa41573039f1ac9e82607b38f1bb931e96"}
Nov 03 13:00:02.734497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.734473    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerStarted","Data":"f65ba936769173e0d4ccaa2c7a13a0621faec1c0b33aca167099a393ae4f21cf"}
Nov 03 13:00:02.736176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.736157    4546 generic.go:356] "Generic (PLEG): container finished" podID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerID="27a90db9574b0f2bc14ab6cb1017da9ff81531f5a287cb58708acbdb30cd9dde" exitCode=60
Nov 03 13:00:02.736282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.736212    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" event={"ID":"4dc7faed-4d52-4e8d-aed3-149fd75522a6","Type":"ContainerDied","Data":"27a90db9574b0f2bc14ab6cb1017da9ff81531f5a287cb58708acbdb30cd9dde"}
Nov 03 13:00:02.736495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.736476    4546 scope.go:122] "RemoveContainer" containerID="27a90db9574b0f2bc14ab6cb1017da9ff81531f5a287cb58708acbdb30cd9dde"
Nov 03 13:00:02.737937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.737914    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerStarted","Data":"b01b55e89ea0ebb714c9c893cc7fb0f8cbf94b681076677c4b808c8b054747b2"}
Nov 03 13:00:02.739584 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.739563    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerStarted","Data":"b128ac749912402d4b7b97a671e613a52e61c9892f161f2cbac4d49d72b9ca72"}
Nov 03 13:00:02.739792 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.739771    4546 scope.go:122] "RemoveContainer" containerID="b128ac749912402d4b7b97a671e613a52e61c9892f161f2cbac4d49d72b9ca72"
Nov 03 13:00:02.741230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.741196    4546 generic.go:356] "Generic (PLEG): container finished" podID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerID="8bd6e4d9d66ff415f1ab291d91422205a2f8b93997ef9cef931211c542168b9e" exitCode=60
Nov 03 13:00:02.741328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.741225    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" event={"ID":"bc292f12-2b0a-4b5a-ac16-416e64c9040c","Type":"ContainerDied","Data":"8bd6e4d9d66ff415f1ab291d91422205a2f8b93997ef9cef931211c542168b9e"}
Nov 03 13:00:02.741493 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.741477    4546 scope.go:122] "RemoveContainer" containerID="8bd6e4d9d66ff415f1ab291d91422205a2f8b93997ef9cef931211c542168b9e"
Nov 03 13:00:02.743313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.743269    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerStarted","Data":"ca1eb63ae1d3be4da43957277f4d64db96dbfb1a38508996a76feb9c03d301c7"}
Nov 03 13:00:02.752300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.751940    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerStarted","Data":"091f2edcffaef5f60221519f0846ff6735b57b7203bbddaef0439ac78ab67132"}
Nov 03 13:00:02.754998 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.754725    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerStarted","Data":"5a92fd78086b0a4adcd525bf6da90d0eea696948241723cfbf56b3be15a6fcc3"}
Nov 03 13:00:02.754998 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.754990    4546 scope.go:122] "RemoveContainer" containerID="5a92fd78086b0a4adcd525bf6da90d0eea696948241723cfbf56b3be15a6fcc3"
Nov 03 13:00:02.756543 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.756445    4546 generic.go:356] "Generic (PLEG): container finished" podID="2d847af6-74a2-45cb-9a6b-74750275610e" containerID="0132ad7d30a6c8651f0f926e41eab1dfff5048a03d78d9b5d9ee3836d8a5d240" exitCode=60
Nov 03 13:00:02.756543 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.756519    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" event={"ID":"2d847af6-74a2-45cb-9a6b-74750275610e","Type":"ContainerDied","Data":"0132ad7d30a6c8651f0f926e41eab1dfff5048a03d78d9b5d9ee3836d8a5d240"}
Nov 03 13:00:02.756777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.756763    4546 scope.go:122] "RemoveContainer" containerID="0132ad7d30a6c8651f0f926e41eab1dfff5048a03d78d9b5d9ee3836d8a5d240"
Nov 03 13:00:02.758234 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.758212    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerStarted","Data":"d8f446ee9c1d8c3035e6c0c44cf85f75c5bfed41279c6de1e14efb1048cf7d98"}
Nov 03 13:00:02.836800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.836729    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" podStartSLOduration=2.836711446 podStartE2EDuration="2.836711446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.833898684 +0000 UTC m=+4144.405482607" watchObservedRunningTime="2026-11-03 13:00:02.836711446 +0000 UTC m=+4144.408295361"
Nov 03 13:00:02.945958 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:02.945904    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" podStartSLOduration=2.945886704 podStartE2EDuration="2.945886704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:02.94131001 +0000 UTC m=+4144.512893946" watchObservedRunningTime="2026-11-03 13:00:02.945886704 +0000 UTC m=+4144.517470635"
Nov 03 13:00:03.359643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.359591    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" podStartSLOduration=3.359575348 podStartE2EDuration="3.359575348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:03.357738337 +0000 UTC m=+4144.929322283" watchObservedRunningTime="2026-11-03 13:00:03.359575348 +0000 UTC m=+4144.931159260"
Nov 03 13:00:03.465250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.465200    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" podStartSLOduration=3.465184908 podStartE2EDuration="3.465184908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:03.461350016 +0000 UTC m=+4145.032933935" watchObservedRunningTime="2026-11-03 13:00:03.465184908 +0000 UTC m=+4145.036768938"
Nov 03 13:00:03.568249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.568199    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" podStartSLOduration=3.568184191 podStartE2EDuration="3.568184191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:03.565120058 +0000 UTC m=+4145.136703989" watchObservedRunningTime="2026-11-03 13:00:03.568184191 +0000 UTC m=+4145.139768104"
Nov 03 13:00:03.766407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.766378    4546 generic.go:356] "Generic (PLEG): container finished" podID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerID="5a92fd78086b0a4adcd525bf6da90d0eea696948241723cfbf56b3be15a6fcc3" exitCode=60
Nov 03 13:00:03.766911 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.766451    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerDied","Data":"5a92fd78086b0a4adcd525bf6da90d0eea696948241723cfbf56b3be15a6fcc3"}
Nov 03 13:00:03.768021 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.768002    4546 generic.go:356] "Generic (PLEG): container finished" podID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerID="d8f446ee9c1d8c3035e6c0c44cf85f75c5bfed41279c6de1e14efb1048cf7d98" exitCode=60
Nov 03 13:00:03.768113 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.768078    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerDied","Data":"d8f446ee9c1d8c3035e6c0c44cf85f75c5bfed41279c6de1e14efb1048cf7d98"}
Nov 03 13:00:03.768376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.768354    4546 scope.go:122] "RemoveContainer" containerID="d8f446ee9c1d8c3035e6c0c44cf85f75c5bfed41279c6de1e14efb1048cf7d98"
Nov 03 13:00:03.770033 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.770013    4546 generic.go:356] "Generic (PLEG): container finished" podID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerID="617d18e571e1bee0d2d403b8d2f8d04479fa9c795b8f12f13b44e111566fff96" exitCode=60
Nov 03 13:00:03.770117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.770065    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerDied","Data":"617d18e571e1bee0d2d403b8d2f8d04479fa9c795b8f12f13b44e111566fff96"}
Nov 03 13:00:03.770328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.770312    4546 scope.go:122] "RemoveContainer" containerID="617d18e571e1bee0d2d403b8d2f8d04479fa9c795b8f12f13b44e111566fff96"
Nov 03 13:00:03.772507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.772425    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerStarted","Data":"94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3"}
Nov 03 13:00:03.774674 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.774650    4546 generic.go:356] "Generic (PLEG): container finished" podID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerID="b8884dba6e9df0fb95323b60a56f1eaa41573039f1ac9e82607b38f1bb931e96" exitCode=60
Nov 03 13:00:03.774824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.774779    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerDied","Data":"b8884dba6e9df0fb95323b60a56f1eaa41573039f1ac9e82607b38f1bb931e96"}
Nov 03 13:00:03.775019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.775004    4546 scope.go:122] "RemoveContainer" containerID="b8884dba6e9df0fb95323b60a56f1eaa41573039f1ac9e82607b38f1bb931e96"
Nov 03 13:00:03.775552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.775442    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" podStartSLOduration=3.775428779 podStartE2EDuration="3.775428779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:03.773223777 +0000 UTC m=+4145.344807689" watchObservedRunningTime="2026-11-03 13:00:03.775428779 +0000 UTC m=+4145.347012690"
Nov 03 13:00:03.776708 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.776688    4546 generic.go:356] "Generic (PLEG): container finished" podID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerID="f65ba936769173e0d4ccaa2c7a13a0621faec1c0b33aca167099a393ae4f21cf" exitCode=60
Nov 03 13:00:03.776830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.776776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerDied","Data":"f65ba936769173e0d4ccaa2c7a13a0621faec1c0b33aca167099a393ae4f21cf"}
Nov 03 13:00:03.776995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.776976    4546 scope.go:122] "RemoveContainer" containerID="f65ba936769173e0d4ccaa2c7a13a0621faec1c0b33aca167099a393ae4f21cf"
Nov 03 13:00:03.779080 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.779052    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" event={"ID":"4dc7faed-4d52-4e8d-aed3-149fd75522a6","Type":"ContainerStarted","Data":"0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2"}
Nov 03 13:00:03.780662 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.780643    4546 generic.go:356] "Generic (PLEG): container finished" podID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerID="b01b55e89ea0ebb714c9c893cc7fb0f8cbf94b681076677c4b808c8b054747b2" exitCode=60
Nov 03 13:00:03.780809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.780739    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerDied","Data":"b01b55e89ea0ebb714c9c893cc7fb0f8cbf94b681076677c4b808c8b054747b2"}
Nov 03 13:00:03.781266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.781245    4546 scope.go:122] "RemoveContainer" containerID="b01b55e89ea0ebb714c9c893cc7fb0f8cbf94b681076677c4b808c8b054747b2"
Nov 03 13:00:03.782809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.782791    4546 generic.go:356] "Generic (PLEG): container finished" podID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerID="b128ac749912402d4b7b97a671e613a52e61c9892f161f2cbac4d49d72b9ca72" exitCode=60
Nov 03 13:00:03.782910 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.782869    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerDied","Data":"b128ac749912402d4b7b97a671e613a52e61c9892f161f2cbac4d49d72b9ca72"}
Nov 03 13:00:03.782976 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.782909    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerStarted","Data":"0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9"}
Nov 03 13:00:03.784693 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.784671    4546 generic.go:356] "Generic (PLEG): container finished" podID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerID="ca1eb63ae1d3be4da43957277f4d64db96dbfb1a38508996a76feb9c03d301c7" exitCode=60
Nov 03 13:00:03.784819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.784785    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerDied","Data":"ca1eb63ae1d3be4da43957277f4d64db96dbfb1a38508996a76feb9c03d301c7"}
Nov 03 13:00:03.784988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.784971    4546 scope.go:122] "RemoveContainer" containerID="ca1eb63ae1d3be4da43957277f4d64db96dbfb1a38508996a76feb9c03d301c7"
Nov 03 13:00:03.786634 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.786614    4546 generic.go:356] "Generic (PLEG): container finished" podID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerID="091f2edcffaef5f60221519f0846ff6735b57b7203bbddaef0439ac78ab67132" exitCode=60
Nov 03 13:00:03.786725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.786654    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerDied","Data":"091f2edcffaef5f60221519f0846ff6735b57b7203bbddaef0439ac78ab67132"}
Nov 03 13:00:03.786932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.786917    4546 scope.go:122] "RemoveContainer" containerID="091f2edcffaef5f60221519f0846ff6735b57b7203bbddaef0439ac78ab67132"
Nov 03 13:00:03.884356 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:03.884312    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" podStartSLOduration=3.884298952 podStartE2EDuration="3.884298952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:03.883176649 +0000 UTC m=+4145.454760561" watchObservedRunningTime="2026-11-03 13:00:03.884298952 +0000 UTC m=+4145.455882863"
Nov 03 13:00:04.303590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.303535    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" podStartSLOduration=4.303521581 podStartE2EDuration="4.303521581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:04.300193502 +0000 UTC m=+4145.871777446" watchObservedRunningTime="2026-11-03 13:00:04.303521581 +0000 UTC m=+4145.875105491"
Nov 03 13:00:04.802680 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.802642    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerStarted","Data":"8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b"}
Nov 03 13:00:04.804643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.804619    4546 generic.go:356] "Generic (PLEG): container finished" podID="201264ad-f94b-412c-ba91-24f933679a7e" containerID="94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3" exitCode=60
Nov 03 13:00:04.804799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.804691    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerDied","Data":"94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3"}
Nov 03 13:00:04.804799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.804761    4546 scope.go:122] "RemoveContainer" containerID="6d014b66c3a0b6b86732943e5d0686baec35cbd1eefdb1fdccfda03f6ecb93bd"
Nov 03 13:00:04.805001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.804972    4546 scope.go:122] "RemoveContainer" containerID="94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3"
Nov 03 13:00:04.805314 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:04.805290    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp_odh-ai-gateway-infra(201264ad-f94b-412c-ba91-24f933679a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" podUID="201264ad-f94b-412c-ba91-24f933679a7e"
Nov 03 13:00:04.808213 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.807439    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerStarted","Data":"725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335"}
Nov 03 13:00:04.814013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.813920    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerStarted","Data":"22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871"}
Nov 03 13:00:04.816544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.816522    4546 generic.go:356] "Generic (PLEG): container finished" podID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerID="0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2" exitCode=60
Nov 03 13:00:04.816653 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.816597    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" event={"ID":"4dc7faed-4d52-4e8d-aed3-149fd75522a6","Type":"ContainerDied","Data":"0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2"}
Nov 03 13:00:04.816849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.816832    4546 scope.go:122] "RemoveContainer" containerID="0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2"
Nov 03 13:00:04.817133 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:04.817108    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth_odh-ai-gateway-infra(4dc7faed-4d52-4e8d-aed3-149fd75522a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6"
Nov 03 13:00:04.818869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.818842    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerStarted","Data":"23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0"}
Nov 03 13:00:04.820847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.820826    4546 generic.go:356] "Generic (PLEG): container finished" podID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerID="0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9" exitCode=60
Nov 03 13:00:04.820941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.820891    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerDied","Data":"0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9"}
Nov 03 13:00:04.821104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.821087    4546 scope.go:122] "RemoveContainer" containerID="0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9"
Nov 03 13:00:04.821364 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:04.821345    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl_odh-ai-gateway-infra(0aaace4b-7c89-47ad-be3e-9582d747f3dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd"
Nov 03 13:00:04.821864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.821843    4546 scope.go:122] "RemoveContainer" containerID="27a90db9574b0f2bc14ab6cb1017da9ff81531f5a287cb58708acbdb30cd9dde"
Nov 03 13:00:04.822787 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.822726    4546 generic.go:356] "Generic (PLEG): container finished" podID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerID="8394c7c52f32ab2907b8dade7c5bac5233fef31af35c5b0c0d96270bc4740800" exitCode=60
Nov 03 13:00:04.822890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.822796    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" event={"ID":"bc292f12-2b0a-4b5a-ac16-416e64c9040c","Type":"ContainerDied","Data":"8394c7c52f32ab2907b8dade7c5bac5233fef31af35c5b0c0d96270bc4740800"}
Nov 03 13:00:04.823168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.823049    4546 scope.go:122] "RemoveContainer" containerID="8394c7c52f32ab2907b8dade7c5bac5233fef31af35c5b0c0d96270bc4740800"
Nov 03 13:00:04.823317 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:04.823291    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4_odh-ai-gateway-infra(bc292f12-2b0a-4b5a-ac16-416e64c9040c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c"
Nov 03 13:00:04.825169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.825149    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerStarted","Data":"eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d"}
Nov 03 13:00:04.825989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.825954    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" podStartSLOduration=4.825942862 podStartE2EDuration="4.825942862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:04.823165068 +0000 UTC m=+4146.394749004" watchObservedRunningTime="2026-11-03 13:00:04.825942862 +0000 UTC m=+4146.397526774"
Nov 03 13:00:04.827150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.827127    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerStarted","Data":"f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e"}
Nov 03 13:00:04.829408 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.829387    4546 generic.go:356] "Generic (PLEG): container finished" podID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerID="5aa0d3c1c846d55a4520dc7339c5193fc04c90745e7294a4b9fb54262cd15589" exitCode=60
Nov 03 13:00:04.829508 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.829460    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerDied","Data":"5aa0d3c1c846d55a4520dc7339c5193fc04c90745e7294a4b9fb54262cd15589"}
Nov 03 13:00:04.829674 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.829659    4546 scope.go:122] "RemoveContainer" containerID="5aa0d3c1c846d55a4520dc7339c5193fc04c90745e7294a4b9fb54262cd15589"
Nov 03 13:00:04.829928 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:04.829905    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j_odh-ai-gateway-infra(b9aa90ac-39fd-4fa8-8f45-bba34671884f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f"
Nov 03 13:00:04.831643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.831623    4546 generic.go:356] "Generic (PLEG): container finished" podID="2d847af6-74a2-45cb-9a6b-74750275610e" containerID="650b09c783d3a604a2b9fe06f7a964a0e42b15992fc651e5e9c879f833807669" exitCode=60
Nov 03 13:00:04.831787 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.831690    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" event={"ID":"2d847af6-74a2-45cb-9a6b-74750275610e","Type":"ContainerDied","Data":"650b09c783d3a604a2b9fe06f7a964a0e42b15992fc651e5e9c879f833807669"}
Nov 03 13:00:04.831957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.831939    4546 scope.go:122] "RemoveContainer" containerID="650b09c783d3a604a2b9fe06f7a964a0e42b15992fc651e5e9c879f833807669"
Nov 03 13:00:04.832147 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:04.832120    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht_odh-ai-gateway-infra(2d847af6-74a2-45cb-9a6b-74750275610e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" podUID="2d847af6-74a2-45cb-9a6b-74750275610e"
Nov 03 13:00:04.833831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.833810    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerStarted","Data":"8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7"}
Nov 03 13:00:04.837531 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.837509    4546 scope.go:122] "RemoveContainer" containerID="b128ac749912402d4b7b97a671e613a52e61c9892f161f2cbac4d49d72b9ca72"
Nov 03 13:00:04.850739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.850710    4546 scope.go:122] "RemoveContainer" containerID="8bd6e4d9d66ff415f1ab291d91422205a2f8b93997ef9cef931211c542168b9e"
Nov 03 13:00:04.864344 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.864327    4546 scope.go:122] "RemoveContainer" containerID="5a92fd78086b0a4adcd525bf6da90d0eea696948241723cfbf56b3be15a6fcc3"
Nov 03 13:00:04.877872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:04.877848    4546 scope.go:122] "RemoveContainer" containerID="0132ad7d30a6c8651f0f926e41eab1dfff5048a03d78d9b5d9ee3836d8a5d240"
Nov 03 13:00:05.843208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.843181    4546 scope.go:122] "RemoveContainer" containerID="5aa0d3c1c846d55a4520dc7339c5193fc04c90745e7294a4b9fb54262cd15589"
Nov 03 13:00:05.843674 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.843552    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j_odh-ai-gateway-infra(b9aa90ac-39fd-4fa8-8f45-bba34671884f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f"
Nov 03 13:00:05.844789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.844768    4546 scope.go:122] "RemoveContainer" containerID="650b09c783d3a604a2b9fe06f7a964a0e42b15992fc651e5e9c879f833807669"
Nov 03 13:00:05.845094 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.845073    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht_odh-ai-gateway-infra(2d847af6-74a2-45cb-9a6b-74750275610e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" podUID="2d847af6-74a2-45cb-9a6b-74750275610e"
Nov 03 13:00:05.846328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.846310    4546 generic.go:356] "Generic (PLEG): container finished" podID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerID="8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7" exitCode=60
Nov 03 13:00:05.846406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.846386    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerDied","Data":"8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7"}
Nov 03 13:00:05.846480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.846467    4546 scope.go:122] "RemoveContainer" containerID="d8f446ee9c1d8c3035e6c0c44cf85f75c5bfed41279c6de1e14efb1048cf7d98"
Nov 03 13:00:05.846618 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.846596    4546 scope.go:122] "RemoveContainer" containerID="8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7"
Nov 03 13:00:05.846934 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.846910    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf_odh-ai-gateway-infra(c11e4db1-4807-411e-a3dc-cf95d30ddd22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22"
Nov 03 13:00:05.848329 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.848311    4546 generic.go:356] "Generic (PLEG): container finished" podID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerID="8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b" exitCode=60
Nov 03 13:00:05.848407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.848364    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerDied","Data":"8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b"}
Nov 03 13:00:05.848617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.848601    4546 scope.go:122] "RemoveContainer" containerID="8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b"
Nov 03 13:00:05.848875 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.848851    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb_odh-ai-gateway-infra(4ad320d7-8fd6-421c-afde-25c47719c2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9"
Nov 03 13:00:05.850076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.850062    4546 scope.go:122] "RemoveContainer" containerID="94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3"
Nov 03 13:00:05.850284 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.850263    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp_odh-ai-gateway-infra(201264ad-f94b-412c-ba91-24f933679a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" podUID="201264ad-f94b-412c-ba91-24f933679a7e"
Nov 03 13:00:05.851874 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.851855    4546 generic.go:356] "Generic (PLEG): container finished" podID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerID="725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335" exitCode=60
Nov 03 13:00:05.851969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.851934    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerDied","Data":"725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335"}
Nov 03 13:00:05.852141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.852126    4546 scope.go:122] "RemoveContainer" containerID="725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335"
Nov 03 13:00:05.852361 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.852342    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz_odh-ai-gateway-infra(1108ffa1-92ac-43f3-95c5-43d0fae9beed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed"
Nov 03 13:00:05.853798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.853782    4546 generic.go:356] "Generic (PLEG): container finished" podID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerID="22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871" exitCode=60
Nov 03 13:00:05.853896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.853833    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerDied","Data":"22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871"}
Nov 03 13:00:05.854038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.854021    4546 scope.go:122] "RemoveContainer" containerID="22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871"
Nov 03 13:00:05.854245 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.854225    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl_odh-ai-gateway-infra(c496b387-af0f-4d62-a5ca-e0029eea7f5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f"
Nov 03 13:00:05.855669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.855652    4546 scope.go:122] "RemoveContainer" containerID="0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2"
Nov 03 13:00:05.855895 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.855869    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth_odh-ai-gateway-infra(4dc7faed-4d52-4e8d-aed3-149fd75522a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6"
Nov 03 13:00:05.857369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.857349    4546 generic.go:356] "Generic (PLEG): container finished" podID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerID="23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0" exitCode=60
Nov 03 13:00:05.857443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.857419    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerDied","Data":"23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0"}
Nov 03 13:00:05.857576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.857562    4546 scope.go:122] "RemoveContainer" containerID="23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0"
Nov 03 13:00:05.857771 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.857737    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc_odh-ai-gateway-infra(39e1e24d-d7ef-4b81-a640-ed4cd8733dcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf"
Nov 03 13:00:05.859265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.859246    4546 scope.go:122] "RemoveContainer" containerID="0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9"
Nov 03 13:00:05.859485 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.859459    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl_odh-ai-gateway-infra(0aaace4b-7c89-47ad-be3e-9582d747f3dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd"
Nov 03 13:00:05.860899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.860881    4546 scope.go:122] "RemoveContainer" containerID="8394c7c52f32ab2907b8dade7c5bac5233fef31af35c5b0c0d96270bc4740800"
Nov 03 13:00:05.861149 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.861126    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4_odh-ai-gateway-infra(bc292f12-2b0a-4b5a-ac16-416e64c9040c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c"
Nov 03 13:00:05.862679 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.862584    4546 scope.go:122] "RemoveContainer" containerID="617d18e571e1bee0d2d403b8d2f8d04479fa9c795b8f12f13b44e111566fff96"
Nov 03 13:00:05.862679 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.862641    4546 generic.go:356] "Generic (PLEG): container finished" podID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerID="eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d" exitCode=60
Nov 03 13:00:05.862994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.862699    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerDied","Data":"eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d"}
Nov 03 13:00:05.862994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.862946    4546 scope.go:122] "RemoveContainer" containerID="eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d"
Nov 03 13:00:05.863184 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.863161    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895180-v25ql_odh-ai-gateway-infra(118b48b1-2a8b-4ec7-aad7-86d195fa586e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e"
Nov 03 13:00:05.865027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.865009    4546 generic.go:356] "Generic (PLEG): container finished" podID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerID="f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e" exitCode=60
Nov 03 13:00:05.865131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.865058    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerDied","Data":"f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e"}
Nov 03 13:00:05.865307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.865289    4546 scope.go:122] "RemoveContainer" containerID="f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e"
Nov 03 13:00:05.865523 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:05.865502    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt_odh-ai-gateway-infra(4f55a1dd-f311-44ad-9c0c-fcee328c98a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3"
Nov 03 13:00:05.877152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.873997    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8" podUID="6d82fb07-7cb7-450c-ba71-7a533dc9603b" containerName="payload-pre-processing" containerID="cri-o://a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c" gracePeriod=25
Nov 03 13:00:05.878724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.878701    4546 scope.go:122] "RemoveContainer" containerID="b8884dba6e9df0fb95323b60a56f1eaa41573039f1ac9e82607b38f1bb931e96"
Nov 03 13:00:05.991359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:05.991334    4546 scope.go:122] "RemoveContainer" containerID="f65ba936769173e0d4ccaa2c7a13a0621faec1c0b33aca167099a393ae4f21cf"
Nov 03 13:00:06.007031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.007010    4546 scope.go:122] "RemoveContainer" containerID="b01b55e89ea0ebb714c9c893cc7fb0f8cbf94b681076677c4b808c8b054747b2"
Nov 03 13:00:06.022139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.022049    4546 scope.go:122] "RemoveContainer" containerID="ca1eb63ae1d3be4da43957277f4d64db96dbfb1a38508996a76feb9c03d301c7"
Nov 03 13:00:06.038594 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.038571    4546 scope.go:122] "RemoveContainer" containerID="091f2edcffaef5f60221519f0846ff6735b57b7203bbddaef0439ac78ab67132"
Nov 03 13:00:06.171124 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.171097    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 13:00:06.284671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.284641    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k\" (UniqueName: \"kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k\") pod \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") "
Nov 03 13:00:06.284938 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.284924    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume\") pod \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\" (UID: \"6d82fb07-7cb7-450c-ba71-7a533dc9603b\") "
Nov 03 13:00:06.285294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.285268    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume" pod "6d82fb07-7cb7-450c-ba71-7a533dc9603b" (UID: "6d82fb07-7cb7-450c-ba71-7a533dc9603b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:06.285423 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.285389    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d82fb07-7cb7-450c-ba71-7a533dc9603b-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:06.286975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.286953    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k" pod "6d82fb07-7cb7-450c-ba71-7a533dc9603b" (UID: "6d82fb07-7cb7-450c-ba71-7a533dc9603b"). InnerVolumeSpecName "kube-api-access-rb55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:06.386581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.386486    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb55k\" (UniqueName: \"kubernetes.io/projected/6d82fb07-7cb7-450c-ba71-7a533dc9603b-kube-api-access-rb55k\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:06.876919 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.876881    4546 scope.go:122] "RemoveContainer" containerID="8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b"
Nov 03 13:00:06.877461 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.877328    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb_odh-ai-gateway-infra(4ad320d7-8fd6-421c-afde-25c47719c2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9"
Nov 03 13:00:06.878680 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.878660    4546 scope.go:122] "RemoveContainer" containerID="725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335"
Nov 03 13:00:06.879002 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.878979    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz_odh-ai-gateway-infra(1108ffa1-92ac-43f3-95c5-43d0fae9beed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed"
Nov 03 13:00:06.880922 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.880895    4546 scope.go:122] "RemoveContainer" containerID="22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871"
Nov 03 13:00:06.881234 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.881210    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl_odh-ai-gateway-infra(c496b387-af0f-4d62-a5ca-e0029eea7f5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f"
Nov 03 13:00:06.882802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.882731    4546 scope.go:122] "RemoveContainer" containerID="23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0"
Nov 03 13:00:06.883062 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.883032    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc_odh-ai-gateway-infra(39e1e24d-d7ef-4b81-a640-ed4cd8733dcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf"
Nov 03 13:00:06.885138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.885118    4546 scope.go:122] "RemoveContainer" containerID="eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d"
Nov 03 13:00:06.885423 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.885399    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895180-v25ql_odh-ai-gateway-infra(118b48b1-2a8b-4ec7-aad7-86d195fa586e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e"
Nov 03 13:00:06.886867 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.886847    4546 scope.go:122] "RemoveContainer" containerID="f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e"
Nov 03 13:00:06.887112 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.887083    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt_odh-ai-gateway-infra(4f55a1dd-f311-44ad-9c0c-fcee328c98a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3"
Nov 03 13:00:06.888047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.888028    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d82fb07-7cb7-450c-ba71-7a533dc9603b" containerID="a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c" exitCode=0
Nov 03 13:00:06.888164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.888071    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"
Nov 03 13:00:06.888164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.888101    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8" event={"ID":"6d82fb07-7cb7-450c-ba71-7a533dc9603b","Type":"ContainerDied","Data":"a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c"}
Nov 03 13:00:06.888164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.888148    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8" event={"ID":"6d82fb07-7cb7-450c-ba71-7a533dc9603b","Type":"ContainerDied","Data":"238c117a17c5a4f31717a200535eceb7f3211ae5b31d9c857574dee3402c665f"}
Nov 03 13:00:06.888347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.888174    4546 scope.go:122] "RemoveContainer" containerID="a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c"
Nov 03 13:00:06.890292 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.890270    4546 scope.go:122] "RemoveContainer" containerID="8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7"
Nov 03 13:00:06.890558 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.890529    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf_odh-ai-gateway-infra(c11e4db1-4807-411e-a3dc-cf95d30ddd22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22"
Nov 03 13:00:06.899573 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.899545    4546 scope.go:122] "RemoveContainer" containerID="a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c"
Nov 03 13:00:06.899967 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:06.899944    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c\": container with ID starting with a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c not found: ID does not exist" containerID="a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c"
Nov 03 13:00:06.900034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:06.899974    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c"} err="failed to get container status \"a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c\": rpc error: code = NotFound desc = could not find container \"a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c\": container with ID starting with a0374bb0a77a5c79a0be1e4da7488f7b1523197f9cbd6093bfab35f48224dc2c not found: ID does not exist"
Nov 03 13:00:07.567440 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:07.567406    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:07.663473 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:07.663442    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:00:08.381406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:08.381372    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"]
Nov 03 13:00:08.386394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:08.386368    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-532442-85d47c47b7-q7js8"]
Nov 03 13:00:08.574509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:08.574467    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d82fb07-7cb7-450c-ba71-7a533dc9603b" path="/var/lib/kubelet/pods/6d82fb07-7cb7-450c-ba71-7a533dc9603b/volumes"
Nov 03 13:00:11.045573 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:11.045536    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"]
Nov 03 13:00:15.566344 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:15.566305    4546 scope.go:122] "RemoveContainer" containerID="5aa0d3c1c846d55a4520dc7339c5193fc04c90745e7294a4b9fb54262cd15589"
Nov 03 13:00:15.566815 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:15.566625    4546 scope.go:122] "RemoveContainer" containerID="0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9"
Nov 03 13:00:15.942873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:15.942846    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:00:15.943009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:15.942888    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:00:15.984194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:15.984161    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerStarted","Data":"91a878d7adbedf83285d5f4b265af7107e6a866c3d862a5c6977d756043b4e87"}
Nov 03 13:00:16.046504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.046466    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9" podUID="6e71e1f9-e133-41ad-90a8-52bfffc343db" containerName="payload-pre-processing" containerID="cri-o://8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da" gracePeriod=25
Nov 03 13:00:16.092677 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.092569    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" podStartSLOduration=16.09255144 podStartE2EDuration="16.09255144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:16.088869724 +0000 UTC m=+4157.660453670" watchObservedRunningTime="2026-11-03 13:00:16.09255144 +0000 UTC m=+4157.664135352"
Nov 03 13:00:16.408132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.408091    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 13:00:16.460915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.460833    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs\" (UniqueName: \"kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs\") pod \"6e71e1f9-e133-41ad-90a8-52bfffc343db\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") "
Nov 03 13:00:16.461083 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.460961    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume\") pod \"6e71e1f9-e133-41ad-90a8-52bfffc343db\" (UID: \"6e71e1f9-e133-41ad-90a8-52bfffc343db\") "
Nov 03 13:00:16.461379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.461356    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume" pod "6e71e1f9-e133-41ad-90a8-52bfffc343db" (UID: "6e71e1f9-e133-41ad-90a8-52bfffc343db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:16.461603 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.461588    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e71e1f9-e133-41ad-90a8-52bfffc343db-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:16.463027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.463007    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs" pod "6e71e1f9-e133-41ad-90a8-52bfffc343db" (UID: "6e71e1f9-e133-41ad-90a8-52bfffc343db"). InnerVolumeSpecName "kube-api-access-7w6bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:16.563890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.563857    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w6bs\" (UniqueName: \"kubernetes.io/projected/6e71e1f9-e133-41ad-90a8-52bfffc343db-kube-api-access-7w6bs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:16.576305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.576276    4546 scope.go:122] "RemoveContainer" containerID="22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871"
Nov 03 13:00:16.577249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.577071    4546 scope.go:122] "RemoveContainer" containerID="650b09c783d3a604a2b9fe06f7a964a0e42b15992fc651e5e9c879f833807669"
Nov 03 13:00:16.577249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.577220    4546 scope.go:122] "RemoveContainer" containerID="8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b"
Nov 03 13:00:16.578253 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.578235    4546 scope.go:122] "RemoveContainer" containerID="8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7"
Nov 03 13:00:16.992718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.992689    4546 generic.go:356] "Generic (PLEG): container finished" podID="2cc20d22-0d86-45f9-af45-6af554757811" containerID="b5167c42a5967110250e8e6d867d0ca6ca757db51f436797068051db9d32de98" exitCode=0
Nov 03 13:00:16.992858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.992778    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" event={"ID":"2cc20d22-0d86-45f9-af45-6af554757811","Type":"ContainerDied","Data":"b5167c42a5967110250e8e6d867d0ca6ca757db51f436797068051db9d32de98"}
Nov 03 13:00:16.993251 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.993230    4546 scope.go:122] "RemoveContainer" containerID="b5167c42a5967110250e8e6d867d0ca6ca757db51f436797068051db9d32de98"
Nov 03 13:00:16.994969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.994946    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerStarted","Data":"54ef26d21656367c9c0f3e3c4b4929f0f8760e8a1bee9ccbaff3e89aaed7911f"}
Nov 03 13:00:16.995075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.995019    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:00:16.995146 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.995105    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:00:16.995211 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.995146    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:00:16.997531 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.997507    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerStarted","Data":"556fc917acf920c3a7063ef68a77f76e901be20b779e30f7469fa9e0a848a7ad"}
Nov 03 13:00:16.999535 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.999517    4546 generic.go:356] "Generic (PLEG): container finished" podID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerID="1af043e0d173e3835ede79493428f17bab98557f73096f3e3d9967bc8a25c208" exitCode=60
Nov 03 13:00:16.999636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.999579    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerDied","Data":"1af043e0d173e3835ede79493428f17bab98557f73096f3e3d9967bc8a25c208"}
Nov 03 13:00:16.999636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.999631    4546 scope.go:122] "RemoveContainer" containerID="0ac658134b5b2a155d8a4af86c2a8fe77c76f6aaa53d6775a264b97a3c63abe9"
Nov 03 13:00:16.999995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:16.999973    4546 scope.go:122] "RemoveContainer" containerID="1af043e0d173e3835ede79493428f17bab98557f73096f3e3d9967bc8a25c208"
Nov 03 13:00:17.000276 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:17.000250    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl_odh-ai-gateway-infra(0aaace4b-7c89-47ad-be3e-9582d747f3dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd"
Nov 03 13:00:17.001140 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.001118    4546 generic.go:356] "Generic (PLEG): container finished" podID="6e71e1f9-e133-41ad-90a8-52bfffc343db" containerID="8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da" exitCode=0
Nov 03 13:00:17.001225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.001180    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"
Nov 03 13:00:17.001225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.001204    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9" event={"ID":"6e71e1f9-e133-41ad-90a8-52bfffc343db","Type":"ContainerDied","Data":"8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da"}
Nov 03 13:00:17.001326 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.001241    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9" event={"ID":"6e71e1f9-e133-41ad-90a8-52bfffc343db","Type":"ContainerDied","Data":"740eb33550d0540f18b86f802012171a5b0575808a4a42e5fcd7c370a838f172"}
Nov 03 13:00:17.003615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.003590    4546 generic.go:356] "Generic (PLEG): container finished" podID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerID="91a878d7adbedf83285d5f4b265af7107e6a866c3d862a5c6977d756043b4e87" exitCode=60
Nov 03 13:00:17.003729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.003636    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerDied","Data":"91a878d7adbedf83285d5f4b265af7107e6a866c3d862a5c6977d756043b4e87"}
Nov 03 13:00:17.003972 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.003956    4546 scope.go:122] "RemoveContainer" containerID="91a878d7adbedf83285d5f4b265af7107e6a866c3d862a5c6977d756043b4e87"
Nov 03 13:00:17.004235 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:17.004214    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j_odh-ai-gateway-infra(b9aa90ac-39fd-4fa8-8f45-bba34671884f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f"
Nov 03 13:00:17.026791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.026652    4546 scope.go:122] "RemoveContainer" containerID="8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da"
Nov 03 13:00:17.055956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.055858    4546 scope.go:122] "RemoveContainer" containerID="8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da"
Nov 03 13:00:17.056211 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:17.056185    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da\": container with ID starting with 8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da not found: ID does not exist" containerID="8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da"
Nov 03 13:00:17.056299 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.056226    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da"} err="failed to get container status \"8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da\": rpc error: code = NotFound desc = could not find container \"8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da\": container with ID starting with 8e2ba76278d9f17d50a24b8ab786367e6484a2f21847c843b9961aff0002a7da not found: ID does not exist"
Nov 03 13:00:17.056299 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.056250    4546 scope.go:122] "RemoveContainer" containerID="5aa0d3c1c846d55a4520dc7339c5193fc04c90745e7294a4b9fb54262cd15589"
Nov 03 13:00:17.102126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.102099    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"]
Nov 03 13:00:17.216172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.216131    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"]
Nov 03 13:00:17.222565 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.222537    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b737fe-9bbb485b7-rssd9"]
Nov 03 13:00:17.565890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.565808    4546 scope.go:122] "RemoveContainer" containerID="23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0"
Nov 03 13:00:17.566070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.565937    4546 scope.go:122] "RemoveContainer" containerID="0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2"
Nov 03 13:00:17.567851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:17.567817    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:18.037793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.037732    4546 generic.go:356] "Generic (PLEG): container finished" podID="2d847af6-74a2-45cb-9a6b-74750275610e" containerID="819ee32b19ea0b95a7f492cbbe145fd06c25ed0cb37aebe31bd9fa7af9a2d7d4" exitCode=60
Nov 03 13:00:18.037793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.037785    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" event={"ID":"2d847af6-74a2-45cb-9a6b-74750275610e","Type":"ContainerDied","Data":"819ee32b19ea0b95a7f492cbbe145fd06c25ed0cb37aebe31bd9fa7af9a2d7d4"}
Nov 03 13:00:18.038249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.037852    4546 scope.go:122] "RemoveContainer" containerID="650b09c783d3a604a2b9fe06f7a964a0e42b15992fc651e5e9c879f833807669"
Nov 03 13:00:18.038394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.038377    4546 scope.go:122] "RemoveContainer" containerID="819ee32b19ea0b95a7f492cbbe145fd06c25ed0cb37aebe31bd9fa7af9a2d7d4"
Nov 03 13:00:18.038805 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:18.038780    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht_odh-ai-gateway-infra(2d847af6-74a2-45cb-9a6b-74750275610e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" podUID="2d847af6-74a2-45cb-9a6b-74750275610e"
Nov 03 13:00:18.042207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.042183    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" event={"ID":"2cc20d22-0d86-45f9-af45-6af554757811","Type":"ContainerStarted","Data":"972ff6d949239b86379984b9ce25369f5b712a548195d2e321f8c3fd08f141f4"}
Nov 03 13:00:18.044186 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.044165    4546 generic.go:356] "Generic (PLEG): container finished" podID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerID="16a5379f2cb935e176263a5e9a78a73ef3d659485adafa234289b91c368909e9" exitCode=60
Nov 03 13:00:18.044299 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.044198    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerDied","Data":"16a5379f2cb935e176263a5e9a78a73ef3d659485adafa234289b91c368909e9"}
Nov 03 13:00:18.044600 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.044584    4546 scope.go:122] "RemoveContainer" containerID="16a5379f2cb935e176263a5e9a78a73ef3d659485adafa234289b91c368909e9"
Nov 03 13:00:18.044956 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:18.044922    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf_odh-ai-gateway-infra(c11e4db1-4807-411e-a3dc-cf95d30ddd22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22"
Nov 03 13:00:18.046469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.046449    4546 generic.go:356] "Generic (PLEG): container finished" podID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerID="54ef26d21656367c9c0f3e3c4b4929f0f8760e8a1bee9ccbaff3e89aaed7911f" exitCode=60
Nov 03 13:00:18.046572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.046516    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerDied","Data":"54ef26d21656367c9c0f3e3c4b4929f0f8760e8a1bee9ccbaff3e89aaed7911f"}
Nov 03 13:00:18.046866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.046849    4546 scope.go:122] "RemoveContainer" containerID="54ef26d21656367c9c0f3e3c4b4929f0f8760e8a1bee9ccbaff3e89aaed7911f"
Nov 03 13:00:18.047091 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:18.047071    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb_odh-ai-gateway-infra(4ad320d7-8fd6-421c-afde-25c47719c2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9"
Nov 03 13:00:18.048727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.048708    4546 generic.go:356] "Generic (PLEG): container finished" podID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerID="556fc917acf920c3a7063ef68a77f76e901be20b779e30f7469fa9e0a848a7ad" exitCode=60
Nov 03 13:00:18.048846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.048776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerDied","Data":"556fc917acf920c3a7063ef68a77f76e901be20b779e30f7469fa9e0a848a7ad"}
Nov 03 13:00:18.049057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.049042    4546 scope.go:122] "RemoveContainer" containerID="556fc917acf920c3a7063ef68a77f76e901be20b779e30f7469fa9e0a848a7ad"
Nov 03 13:00:18.049303 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:18.049280    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl_odh-ai-gateway-infra(c496b387-af0f-4d62-a5ca-e0029eea7f5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f"
Nov 03 13:00:18.054563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.054544    4546 scope.go:122] "RemoveContainer" containerID="8b756aa3f836cc31ed3ea35ffb3bb622dc6638b7b28ae005e4a4a99b81811bb7"
Nov 03 13:00:18.068176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.068110    4546 scope.go:122] "RemoveContainer" containerID="8a553c914c6d0cd9ea0df448d044beda6da05b5884bf48334822dd3a122aae7b"
Nov 03 13:00:18.089641 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.082465    4546 scope.go:122] "RemoveContainer" containerID="22f9aa47f0778233379824a99ac59df7e9e4d7e81fbdadf069af9a6c5435a871"
Nov 03 13:00:18.134888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.134848    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 13:00:18.135205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.135181    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 03 13:00:18.135285 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.135232    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 03 13:00:18.262028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.261999    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:18.338625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.338594    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"]
Nov 03 13:00:18.390903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.390861    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca\") pod \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") "
Nov 03 13:00:18.391125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.391091    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c\" (UniqueName: \"kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c\") pod \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\" (UID: \"b9aa90ac-39fd-4fa8-8f45-bba34671884f\") "
Nov 03 13:00:18.391237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.391181    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca" pod "b9aa90ac-39fd-4fa8-8f45-bba34671884f" (UID: "b9aa90ac-39fd-4fa8-8f45-bba34671884f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:18.391850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.391831    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9aa90ac-39fd-4fa8-8f45-bba34671884f-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:18.393254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.393233    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c" pod "b9aa90ac-39fd-4fa8-8f45-bba34671884f" (UID: "b9aa90ac-39fd-4fa8-8f45-bba34671884f"). InnerVolumeSpecName "kube-api-access-vns4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:18.449879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.449824    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"]
Nov 03 13:00:18.493166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.493123    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vns4c\" (UniqueName: \"kubernetes.io/projected/b9aa90ac-39fd-4fa8-8f45-bba34671884f-kube-api-access-vns4c\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:18.505429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.505400    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:18.566197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.566156    4546 scope.go:122] "RemoveContainer" containerID="94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3"
Nov 03 13:00:18.566379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.566330    4546 scope.go:122] "RemoveContainer" containerID="f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e"
Nov 03 13:00:18.566793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.566740    4546 scope.go:122] "RemoveContainer" containerID="725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335"
Nov 03 13:00:18.575982 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.575957    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e71e1f9-e133-41ad-90a8-52bfffc343db" path="/var/lib/kubelet/pods/6e71e1f9-e133-41ad-90a8-52bfffc343db/volumes"
Nov 03 13:00:18.594409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.594369    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca\") pod \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") "
Nov 03 13:00:18.598967 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.595660    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh\" (UniqueName: \"kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh\") pod \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\" (UID: \"0aaace4b-7c89-47ad-be3e-9582d747f3dd\") "
Nov 03 13:00:18.598967 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.597208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca" pod "0aaace4b-7c89-47ad-be3e-9582d747f3dd" (UID: "0aaace4b-7c89-47ad-be3e-9582d747f3dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:18.602119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.602083    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh" pod "0aaace4b-7c89-47ad-be3e-9582d747f3dd" (UID: "0aaace4b-7c89-47ad-be3e-9582d747f3dd"). InnerVolumeSpecName "kube-api-access-zdkwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:18.651135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.651108    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"]
Nov 03 13:00:18.697668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.697636    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aaace4b-7c89-47ad-be3e-9582d747f3dd-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:18.697805 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:18.697672    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkwh\" (UniqueName: \"kubernetes.io/projected/0aaace4b-7c89-47ad-be3e-9582d747f3dd-kube-api-access-zdkwh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.063568 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.063538    4546 generic.go:356] "Generic (PLEG): container finished" podID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerID="d6f3976f2e3c5d0bdaeaee559b65244f6ff4a7c42df9f354622fb858ef875dc5" exitCode=60
Nov 03 13:00:19.064005 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.063616    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" event={"ID":"4dc7faed-4d52-4e8d-aed3-149fd75522a6","Type":"ContainerDied","Data":"d6f3976f2e3c5d0bdaeaee559b65244f6ff4a7c42df9f354622fb858ef875dc5"}
Nov 03 13:00:19.064005 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.063689    4546 scope.go:122] "RemoveContainer" containerID="0ecbdd2719c0df66b9638f0f08af428118e1725cf95bca4740653b65bae03ec2"
Nov 03 13:00:19.064271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.064254    4546 scope.go:122] "RemoveContainer" containerID="d6f3976f2e3c5d0bdaeaee559b65244f6ff4a7c42df9f354622fb858ef875dc5"
Nov 03 13:00:19.064798 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:19.064776    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth_odh-ai-gateway-infra(4dc7faed-4d52-4e8d-aed3-149fd75522a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6"
Nov 03 13:00:19.065777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.065740    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerStarted","Data":"6fc0f2b12dcdb56947cf1cdfa6c0af0a6e53b7ad4300abcc1e745402de05a4c8"}
Nov 03 13:00:19.067129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.067111    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"
Nov 03 13:00:19.067237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.067149    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl" event={"ID":"0aaace4b-7c89-47ad-be3e-9582d747f3dd","Type":"ContainerDied","Data":"3dcd1f191a8d79d14eaea5287227b2912efd0fd1fac3f9eda82c871f69339de3"}
Nov 03 13:00:19.068684 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.068665    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"
Nov 03 13:00:19.068855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.068683    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j" event={"ID":"b9aa90ac-39fd-4fa8-8f45-bba34671884f","Type":"ContainerDied","Data":"f57c97716f70f45fd30ca7345e37e977ddd7aa02ea553265a53f3c7aaecdf0a7"}
Nov 03 13:00:19.136778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.135342    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 03 13:00:19.136778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.135414    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 03 13:00:19.163506 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.163477    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"]
Nov 03 13:00:19.251987 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.251896    4546 scope.go:122] "RemoveContainer" containerID="1af043e0d173e3835ede79493428f17bab98557f73096f3e3d9967bc8a25c208"
Nov 03 13:00:19.269432 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.269404    4546 scope.go:122] "RemoveContainer" containerID="91a878d7adbedf83285d5f4b265af7107e6a866c3d862a5c6977d756043b4e87"
Nov 03 13:00:19.357882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.357857    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:19.388969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.388892    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"]
Nov 03 13:00:19.432781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.429292    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:19.452048 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.451550    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca\") pod \"4ad320d7-8fd6-421c-afde-25c47719c2d9\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") "
Nov 03 13:00:19.452244 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.451979    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca" pod "4ad320d7-8fd6-421c-afde-25c47719c2d9" (UID: "4ad320d7-8fd6-421c-afde-25c47719c2d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:19.452244 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.452198    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7\" (UniqueName: \"kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7\") pod \"4ad320d7-8fd6-421c-afde-25c47719c2d9\" (UID: \"4ad320d7-8fd6-421c-afde-25c47719c2d9\") "
Nov 03 13:00:19.453455 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.453428    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad320d7-8fd6-421c-afde-25c47719c2d9-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.455206 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.455168    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7" pod "4ad320d7-8fd6-421c-afde-25c47719c2d9" (UID: "4ad320d7-8fd6-421c-afde-25c47719c2d9"). InnerVolumeSpecName "kube-api-access-rknx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:19.475923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.475895    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:19.554826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.554792    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25\" (UniqueName: \"kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25\") pod \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") "
Nov 03 13:00:19.555007 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.554917    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca\") pod \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\" (UID: \"c496b387-af0f-4d62-a5ca-e0029eea7f5f\") "
Nov 03 13:00:19.555579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.555558    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rknx7\" (UniqueName: \"kubernetes.io/projected/4ad320d7-8fd6-421c-afde-25c47719c2d9-kube-api-access-rknx7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.555654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.555607    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca" pod "c496b387-af0f-4d62-a5ca-e0029eea7f5f" (UID: "c496b387-af0f-4d62-a5ca-e0029eea7f5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:19.560215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.558153    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25" pod "c496b387-af0f-4d62-a5ca-e0029eea7f5f" (UID: "c496b387-af0f-4d62-a5ca-e0029eea7f5f"). InnerVolumeSpecName "kube-api-access-dvb25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:19.565904 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.565831    4546 scope.go:122] "RemoveContainer" containerID="8394c7c52f32ab2907b8dade7c5bac5233fef31af35c5b0c0d96270bc4740800"
Nov 03 13:00:19.566168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.566148    4546 scope.go:122] "RemoveContainer" containerID="eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d"
Nov 03 13:00:19.656872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.656530    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca\") pod \"2d847af6-74a2-45cb-9a6b-74750275610e\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") "
Nov 03 13:00:19.656872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.656843    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca" pod "2d847af6-74a2-45cb-9a6b-74750275610e" (UID: "2d847af6-74a2-45cb-9a6b-74750275610e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:19.658306 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.657201    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq\" (UniqueName: \"kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq\") pod \"2d847af6-74a2-45cb-9a6b-74750275610e\" (UID: \"2d847af6-74a2-45cb-9a6b-74750275610e\") "
Nov 03 13:00:19.659628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.659404    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d847af6-74a2-45cb-9a6b-74750275610e-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.659628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.659430    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvb25\" (UniqueName: \"kubernetes.io/projected/c496b387-af0f-4d62-a5ca-e0029eea7f5f-kube-api-access-dvb25\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.659628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.659440    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c496b387-af0f-4d62-a5ca-e0029eea7f5f-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.660363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.660331    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq" pod "2d847af6-74a2-45cb-9a6b-74750275610e" (UID: "2d847af6-74a2-45cb-9a6b-74750275610e"). InnerVolumeSpecName "kube-api-access-kmknq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:19.740033 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.715922    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:19.760709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.760680    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmknq\" (UniqueName: \"kubernetes.io/projected/2d847af6-74a2-45cb-9a6b-74750275610e-kube-api-access-kmknq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.879781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.863538    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5\" (UniqueName: \"kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5\") pod \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") "
Nov 03 13:00:19.879781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.863605    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca\") pod \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\" (UID: \"c11e4db1-4807-411e-a3dc-cf95d30ddd22\") "
Nov 03 13:00:19.879781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.866289    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca" pod "c11e4db1-4807-411e-a3dc-cf95d30ddd22" (UID: "c11e4db1-4807-411e-a3dc-cf95d30ddd22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:19.947054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.928907    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5" pod "c11e4db1-4807-411e-a3dc-cf95d30ddd22" (UID: "c11e4db1-4807-411e-a3dc-cf95d30ddd22"). InnerVolumeSpecName "kube-api-access-q5cr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:19.975796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.975584    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5cr5\" (UniqueName: \"kubernetes.io/projected/c11e4db1-4807-411e-a3dc-cf95d30ddd22-kube-api-access-q5cr5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:19.975796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:19.975632    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11e4db1-4807-411e-a3dc-cf95d30ddd22-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.097988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.097946    4546 generic.go:356] "Generic (PLEG): container finished" podID="201264ad-f94b-412c-ba91-24f933679a7e" containerID="9dd4f8c928470cdee3f1688ad4696948adee021f3edbe9ce56cf01c6b09f548f" exitCode=60
Nov 03 13:00:20.098528 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.098053    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerDied","Data":"9dd4f8c928470cdee3f1688ad4696948adee021f3edbe9ce56cf01c6b09f548f"}
Nov 03 13:00:20.098528 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.098110    4546 scope.go:122] "RemoveContainer" containerID="94cfaf0a12a3e131309feccba0dcf3e7657b47e1ab0843dedfef5c2c509bcec3"
Nov 03 13:00:20.098653 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.098565    4546 scope.go:122] "RemoveContainer" containerID="9dd4f8c928470cdee3f1688ad4696948adee021f3edbe9ce56cf01c6b09f548f"
Nov 03 13:00:20.099004 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:20.098976    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp_odh-ai-gateway-infra(201264ad-f94b-412c-ba91-24f933679a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" podUID="201264ad-f94b-412c-ba91-24f933679a7e"
Nov 03 13:00:20.101724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.101696    4546 generic.go:356] "Generic (PLEG): container finished" podID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerID="da042cbf02064e7dd6e987e27263578e12d018ea683d298a1bf9c1f17b0dfbe8" exitCode=60
Nov 03 13:00:20.101894 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.101779    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerDied","Data":"da042cbf02064e7dd6e987e27263578e12d018ea683d298a1bf9c1f17b0dfbe8"}
Nov 03 13:00:20.102204 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.102185    4546 scope.go:122] "RemoveContainer" containerID="da042cbf02064e7dd6e987e27263578e12d018ea683d298a1bf9c1f17b0dfbe8"
Nov 03 13:00:20.102579 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:20.102554    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz_odh-ai-gateway-infra(1108ffa1-92ac-43f3-95c5-43d0fae9beed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed"
Nov 03 13:00:20.104293 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.104275    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"
Nov 03 13:00:20.104373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.104272    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl" event={"ID":"c496b387-af0f-4d62-a5ca-e0029eea7f5f","Type":"ContainerDied","Data":"a05a936ea073f6a154e718a4be5e9594caf7aa9118fc93ea94fb9c22a3e4af45"}
Nov 03 13:00:20.137293 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.137208    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 03 13:00:20.137293 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.137264    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 03 13:00:20.141111 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.141085    4546 generic.go:356] "Generic (PLEG): container finished" podID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerID="6fc0f2b12dcdb56947cf1cdfa6c0af0a6e53b7ad4300abcc1e745402de05a4c8" exitCode=60
Nov 03 13:00:20.141487 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.141208    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerDied","Data":"6fc0f2b12dcdb56947cf1cdfa6c0af0a6e53b7ad4300abcc1e745402de05a4c8"}
Nov 03 13:00:20.142943 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.142920    4546 scope.go:122] "RemoveContainer" containerID="6fc0f2b12dcdb56947cf1cdfa6c0af0a6e53b7ad4300abcc1e745402de05a4c8"
Nov 03 13:00:20.145113 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:20.143286    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc_odh-ai-gateway-infra(39e1e24d-d7ef-4b81-a640-ed4cd8733dcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf"
Nov 03 13:00:20.147330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.147307    4546 scope.go:122] "RemoveContainer" containerID="725183800f49c7f10c256ea5972e09603e95badd978967acb7c601e6aea15335"
Nov 03 13:00:20.151821 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.151774    4546 generic.go:356] "Generic (PLEG): container finished" podID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerID="83be027c3406a687011b15ef1c0a119ed31ad8413f4cb3b3032cbe595b77f31f" exitCode=60
Nov 03 13:00:20.151955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.151887    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerDied","Data":"83be027c3406a687011b15ef1c0a119ed31ad8413f4cb3b3032cbe595b77f31f"}
Nov 03 13:00:20.152473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.152454    4546 scope.go:122] "RemoveContainer" containerID="83be027c3406a687011b15ef1c0a119ed31ad8413f4cb3b3032cbe595b77f31f"
Nov 03 13:00:20.152884 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:20.152860    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt_odh-ai-gateway-infra(4f55a1dd-f311-44ad-9c0c-fcee328c98a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3"
Nov 03 13:00:20.158638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.155510    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht" event={"ID":"2d847af6-74a2-45cb-9a6b-74750275610e","Type":"ContainerDied","Data":"7b4f73f08d8907d2a310a8a121f1ccfa1862a83c9e2887a65d46a4c3facf80e4"}
Nov 03 13:00:20.158638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.155576    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"
Nov 03 13:00:20.160572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.160545    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf" event={"ID":"c11e4db1-4807-411e-a3dc-cf95d30ddd22","Type":"ContainerDied","Data":"06008f71eb58c46300dbc795222dfd0e8f9163f2c054efe0961326bef4c02261"}
Nov 03 13:00:20.160663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.160645    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"
Nov 03 13:00:20.164739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.164683    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb" event={"ID":"4ad320d7-8fd6-421c-afde-25c47719c2d9","Type":"ContainerDied","Data":"4b87808511e827c541618f281e0a796fd6c8cc7122e64bf4eaa375eccebc5f72"}
Nov 03 13:00:20.164880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.164788    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"
Nov 03 13:00:20.181084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.181063    4546 scope.go:122] "RemoveContainer" containerID="556fc917acf920c3a7063ef68a77f76e901be20b779e30f7469fa9e0a848a7ad"
Nov 03 13:00:20.195313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.195298    4546 scope.go:122] "RemoveContainer" containerID="23c498b1948df5b4bb7446bd08f07fe64c11caaf4990019c341defddc75b81b0"
Nov 03 13:00:20.237665 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.237647    4546 scope.go:122] "RemoveContainer" containerID="f94d76a8d089dc699812ce33d4594f66afb03c70c13a257e8071263eeb21636e"
Nov 03 13:00:20.251408 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.251380    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"]
Nov 03 13:00:20.252995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.252978    4546 scope.go:122] "RemoveContainer" containerID="819ee32b19ea0b95a7f492cbbe145fd06c25ed0cb37aebe31bd9fa7af9a2d7d4"
Nov 03 13:00:20.258467 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.258444    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8570a0-29895180-dd9pl"]
Nov 03 13:00:20.263262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.263246    4546 scope.go:122] "RemoveContainer" containerID="16a5379f2cb935e176263a5e9a78a73ef3d659485adafa234289b91c368909e9"
Nov 03 13:00:20.272636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.272620    4546 scope.go:122] "RemoveContainer" containerID="54ef26d21656367c9c0f3e3c4b4929f0f8760e8a1bee9ccbaff3e89aaed7911f"
Nov 03 13:00:20.328774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.326829    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"]
Nov 03 13:00:20.557544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.557493    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"]
Nov 03 13:00:20.577152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.564491    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895180-r944j"]
Nov 03 13:00:20.593462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.593430    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" path="/var/lib/kubelet/pods/0aaace4b-7c89-47ad-be3e-9582d747f3dd/volumes"
Nov 03 13:00:20.594185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.594169    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" path="/var/lib/kubelet/pods/b9aa90ac-39fd-4fa8-8f45-bba34671884f/volumes"
Nov 03 13:00:20.655378 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.654852    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"]
Nov 03 13:00:20.701855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.701832    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"]
Nov 03 13:00:20.742571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.742538    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b737fe-29895180-kf4pl"]
Nov 03 13:00:20.758334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.758309    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:20.850239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.850207    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca\") pod \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") "
Nov 03 13:00:20.850396 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.850370    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt\" (UniqueName: \"kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt\") pod \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\" (UID: \"4dc7faed-4d52-4e8d-aed3-149fd75522a6\") "
Nov 03 13:00:20.850625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.850603    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca" pod "4dc7faed-4d52-4e8d-aed3-149fd75522a6" (UID: "4dc7faed-4d52-4e8d-aed3-149fd75522a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:20.850811 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.850797    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc7faed-4d52-4e8d-aed3-149fd75522a6-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:20.852579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.852554    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt" pod "4dc7faed-4d52-4e8d-aed3-149fd75522a6" (UID: "4dc7faed-4d52-4e8d-aed3-149fd75522a6"). InnerVolumeSpecName "kube-api-access-h2jpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:20.951707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:20.951676    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jpt\" (UniqueName: \"kubernetes.io/projected/4dc7faed-4d52-4e8d-aed3-149fd75522a6-kube-api-access-h2jpt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.048233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.048085    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"]
Nov 03 13:00:21.054035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.054003    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb74b5-29895180-4mnht"]
Nov 03 13:00:21.135108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.135014    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 03 13:00:21.135108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.135065    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 03 13:00:21.180100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.177481    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"
Nov 03 13:00:21.180100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.177489    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth" event={"ID":"4dc7faed-4d52-4e8d-aed3-149fd75522a6","Type":"ContainerDied","Data":"af62bd23aabed028cb0a982fcd234fa92712cd5042cf6f59cf647e1e6cb11201"}
Nov 03 13:00:21.180100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.177556    4546 scope.go:122] "RemoveContainer" containerID="d6f3976f2e3c5d0bdaeaee559b65244f6ff4a7c42df9f354622fb858ef875dc5"
Nov 03 13:00:21.182388 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.182362    4546 generic.go:356] "Generic (PLEG): container finished" podID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerID="79beb4312c57bee300f55aff512a9d3e5c6f46eb294aaab1fb3ec4680bb12bcd" exitCode=60
Nov 03 13:00:21.182519 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.182448    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" event={"ID":"bc292f12-2b0a-4b5a-ac16-416e64c9040c","Type":"ContainerDied","Data":"79beb4312c57bee300f55aff512a9d3e5c6f46eb294aaab1fb3ec4680bb12bcd"}
Nov 03 13:00:21.182835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.182816    4546 scope.go:122] "RemoveContainer" containerID="79beb4312c57bee300f55aff512a9d3e5c6f46eb294aaab1fb3ec4680bb12bcd"
Nov 03 13:00:21.183092 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:21.183072    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4_odh-ai-gateway-infra(bc292f12-2b0a-4b5a-ac16-416e64c9040c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c"
Nov 03 13:00:21.186629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.185914    4546 generic.go:356] "Generic (PLEG): container finished" podID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerID="fd3462af878671bdd47c733b14f630274bee7e076dfbc02992a5072b2e48f4da" exitCode=60
Nov 03 13:00:21.186629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.186024    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerDied","Data":"fd3462af878671bdd47c733b14f630274bee7e076dfbc02992a5072b2e48f4da"}
Nov 03 13:00:21.186629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.186400    4546 scope.go:122] "RemoveContainer" containerID="fd3462af878671bdd47c733b14f630274bee7e076dfbc02992a5072b2e48f4da"
Nov 03 13:00:21.186865 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:21.186678    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895180-v25ql_odh-ai-gateway-infra(118b48b1-2a8b-4ec7-aad7-86d195fa586e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e"
Nov 03 13:00:21.199069 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.199050    4546 scope.go:122] "RemoveContainer" containerID="8394c7c52f32ab2907b8dade7c5bac5233fef31af35c5b0c0d96270bc4740800"
Nov 03 13:00:21.242381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.242350    4546 scope.go:122] "RemoveContainer" containerID="eefcc477f7ca6a3fd20bdac4a2604147d57075d2e60de22741ba3a37febb053d"
Nov 03 13:00:21.400348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.400259    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:21.402590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.402561    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"]
Nov 03 13:00:21.424786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.413803    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9893f0-29895180-86wkf"]
Nov 03 13:00:21.459939 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.459907    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca\") pod \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") "
Nov 03 13:00:21.460142 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.459993    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5\" (UniqueName: \"kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5\") pod \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\" (UID: \"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf\") "
Nov 03 13:00:21.460334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.460310    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca" pod "39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" (UID: "39e1e24d-d7ef-4b81-a640-ed4cd8733dcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:21.460417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.460405    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.462265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.462245    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5" pod "39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" (UID: "39e1e24d-d7ef-4b81-a640-ed4cd8733dcf"). InnerVolumeSpecName "kube-api-access-2h9n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:21.562036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.561648    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9n5\" (UniqueName: \"kubernetes.io/projected/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf-kube-api-access-2h9n5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:21.601447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.601418    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"]
Nov 03 13:00:21.606739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.606712    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d126a6-29895180-vqjxb"]
Nov 03 13:00:21.947937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:21.947902    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"]
Nov 03 13:00:22.101162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.101123    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"]
Nov 03 13:00:22.137518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.137487    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895180-xsfth"]
Nov 03 13:00:22.137940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.137617    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 03 13:00:22.137940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.137663    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 03 13:00:22.154232 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.154176    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:22.164684 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.164661    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"]
Nov 03 13:00:22.169727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.169670    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca\") pod \"201264ad-f94b-412c-ba91-24f933679a7e\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") "
Nov 03 13:00:22.169845 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.169731    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn\" (UniqueName: \"kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn\") pod \"201264ad-f94b-412c-ba91-24f933679a7e\" (UID: \"201264ad-f94b-412c-ba91-24f933679a7e\") "
Nov 03 13:00:22.170099 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.170072    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca" pod "201264ad-f94b-412c-ba91-24f933679a7e" (UID: "201264ad-f94b-412c-ba91-24f933679a7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:22.170480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.170459    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201264ad-f94b-412c-ba91-24f933679a7e-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.172213 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.172191    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn" pod "201264ad-f94b-412c-ba91-24f933679a7e" (UID: "201264ad-f94b-412c-ba91-24f933679a7e"). InnerVolumeSpecName "kube-api-access-74tjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:22.202368 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.202294    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"
Nov 03 13:00:22.202368 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.202325    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp" event={"ID":"201264ad-f94b-412c-ba91-24f933679a7e","Type":"ContainerDied","Data":"2efeb9fc6d5ba7fa0c61fe8368c99ac795766028fc5f8a563bb53db710c3bc6a"}
Nov 03 13:00:22.202566 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.202401    4546 scope.go:122] "RemoveContainer" containerID="9dd4f8c928470cdee3f1688ad4696948adee021f3edbe9ce56cf01c6b09f548f"
Nov 03 13:00:22.205643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.205621    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc" event={"ID":"39e1e24d-d7ef-4b81-a640-ed4cd8733dcf","Type":"ContainerDied","Data":"6ea34f2843f859bab58040f714c262250f1cb65eec737c25f479f054093019d9"}
Nov 03 13:00:22.205789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.205703    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"
Nov 03 13:00:22.246128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.246106    4546 scope.go:122] "RemoveContainer" containerID="6fc0f2b12dcdb56947cf1cdfa6c0af0a6e53b7ad4300abcc1e745402de05a4c8"
Nov 03 13:00:22.275793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.273256    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74tjn\" (UniqueName: \"kubernetes.io/projected/201264ad-f94b-412c-ba91-24f933679a7e-kube-api-access-74tjn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.275793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.275286    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"]
Nov 03 13:00:22.388372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.388347    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:22.475717 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.475638    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb\" (UniqueName: \"kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb\") pod \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") "
Nov 03 13:00:22.475936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.475805    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca\") pod \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\" (UID: \"1108ffa1-92ac-43f3-95c5-43d0fae9beed\") "
Nov 03 13:00:22.476486 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.476457    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca" pod "1108ffa1-92ac-43f3-95c5-43d0fae9beed" (UID: "1108ffa1-92ac-43f3-95c5-43d0fae9beed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:22.479566 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.479546    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb" pod "1108ffa1-92ac-43f3-95c5-43d0fae9beed" (UID: "1108ffa1-92ac-43f3-95c5-43d0fae9beed"). InnerVolumeSpecName "kube-api-access-9n4zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:22.525997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.524428    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:22.573858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.573828    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" path="/var/lib/kubelet/pods/2d847af6-74a2-45cb-9a6b-74750275610e/volumes"
Nov 03 13:00:22.574318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.574304    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" path="/var/lib/kubelet/pods/4ad320d7-8fd6-421c-afde-25c47719c2d9/volumes"
Nov 03 13:00:22.574692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.574679    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" path="/var/lib/kubelet/pods/4dc7faed-4d52-4e8d-aed3-149fd75522a6/volumes"
Nov 03 13:00:22.575402 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.575386    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" path="/var/lib/kubelet/pods/c11e4db1-4807-411e-a3dc-cf95d30ddd22/volumes"
Nov 03 13:00:22.575820 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.575804    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" path="/var/lib/kubelet/pods/c496b387-af0f-4d62-a5ca-e0029eea7f5f/volumes"
Nov 03 13:00:22.577802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.577781    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9\" (UniqueName: \"kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9\") pod \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") "
Nov 03 13:00:22.577875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.577815    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca\") pod \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\" (UID: \"4f55a1dd-f311-44ad-9c0c-fcee328c98a3\") "
Nov 03 13:00:22.578130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.578117    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n4zb\" (UniqueName: \"kubernetes.io/projected/1108ffa1-92ac-43f3-95c5-43d0fae9beed-kube-api-access-9n4zb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.578187 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.578132    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1108ffa1-92ac-43f3-95c5-43d0fae9beed-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.578411 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.578387    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca" pod "4f55a1dd-f311-44ad-9c0c-fcee328c98a3" (UID: "4f55a1dd-f311-44ad-9c0c-fcee328c98a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:22.580280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.580258    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9" pod "4f55a1dd-f311-44ad-9c0c-fcee328c98a3" (UID: "4f55a1dd-f311-44ad-9c0c-fcee328c98a3"). InnerVolumeSpecName "kube-api-access-6zrx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:22.679629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.679581    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrx9\" (UniqueName: \"kubernetes.io/projected/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-kube-api-access-6zrx9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.679629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.679621    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f55a1dd-f311-44ad-9c0c-fcee328c98a3-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.710771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.708246    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"]
Nov 03 13:00:22.725839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.724805    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91cd99-29895180-zr9pc"]
Nov 03 13:00:22.729773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.728383    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"]
Nov 03 13:00:22.842578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.842473    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"]
Nov 03 13:00:22.846001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.845833    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895180-mtjhp"]
Nov 03 13:00:22.944485 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.944445    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:22.951960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.951930    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"]
Nov 03 13:00:22.986589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.983736    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca\") pod \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") "
Nov 03 13:00:22.986589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.983947    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48\" (UniqueName: \"kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48\") pod \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\" (UID: \"118b48b1-2a8b-4ec7-aad7-86d195fa586e\") "
Nov 03 13:00:22.986589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.984151    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca" pod "118b48b1-2a8b-4ec7-aad7-86d195fa586e" (UID: "118b48b1-2a8b-4ec7-aad7-86d195fa586e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:22.986589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.984722    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118b48b1-2a8b-4ec7-aad7-86d195fa586e-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:22.987208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:22.987182    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48" pod "118b48b1-2a8b-4ec7-aad7-86d195fa586e" (UID: "118b48b1-2a8b-4ec7-aad7-86d195fa586e"). InnerVolumeSpecName "kube-api-access-7dh48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:23.086269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.086240    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dh48\" (UniqueName: \"kubernetes.io/projected/118b48b1-2a8b-4ec7-aad7-86d195fa586e-kube-api-access-7dh48\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:23.136774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.135764    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 03 13:00:23.136774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.135824    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 03 13:00:23.170615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.170589    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:23.187401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.187374    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7\" (UniqueName: \"kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7\") pod \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") "
Nov 03 13:00:23.187575 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.187425    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca\") pod \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\" (UID: \"bc292f12-2b0a-4b5a-ac16-416e64c9040c\") "
Nov 03 13:00:23.188048 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.188025    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca" pod "bc292f12-2b0a-4b5a-ac16-416e64c9040c" (UID: "bc292f12-2b0a-4b5a-ac16-416e64c9040c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:23.189653 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.189634    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7" pod "bc292f12-2b0a-4b5a-ac16-416e64c9040c" (UID: "bc292f12-2b0a-4b5a-ac16-416e64c9040c"). InnerVolumeSpecName "kube-api-access-cf7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:23.253578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.253484    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4" event={"ID":"bc292f12-2b0a-4b5a-ac16-416e64c9040c","Type":"ContainerDied","Data":"a32e0dea084a69928b04622fd1e1511427057fa574b84b07c691f8a3fe918437"}
Nov 03 13:00:23.253578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.253562    4546 scope.go:122] "RemoveContainer" containerID="79beb4312c57bee300f55aff512a9d3e5c6f46eb294aaab1fb3ec4680bb12bcd"
Nov 03 13:00:23.253854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.253648    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"
Nov 03 13:00:23.255268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.255238    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql" event={"ID":"118b48b1-2a8b-4ec7-aad7-86d195fa586e","Type":"ContainerDied","Data":"8718493aefd29c076b6e449a59444aef7517b0b6bc561f9450c46d217e86127c"}
Nov 03 13:00:23.255537 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.255320    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"
Nov 03 13:00:23.258118 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.258097    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"
Nov 03 13:00:23.258271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.258241    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt" event={"ID":"4f55a1dd-f311-44ad-9c0c-fcee328c98a3","Type":"ContainerDied","Data":"457b838780082092f95703b682f1affca144a13ee20d0792f6d04dca7bde7ee4"}
Nov 03 13:00:23.261858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.260668    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz" event={"ID":"1108ffa1-92ac-43f3-95c5-43d0fae9beed","Type":"ContainerDied","Data":"0ed21a4ee61a1787cd97c0521c203050a55ff7e0d63100714b2df5832d6c5939"}
Nov 03 13:00:23.261858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.260781    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"
Nov 03 13:00:23.275277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.274841    4546 scope.go:122] "RemoveContainer" containerID="fd3462af878671bdd47c733b14f630274bee7e076dfbc02992a5072b2e48f4da"
Nov 03 13:00:23.287593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.287569    4546 scope.go:122] "RemoveContainer" containerID="83be027c3406a687011b15ef1c0a119ed31ad8413f4cb3b3032cbe595b77f31f"
Nov 03 13:00:23.289050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.289021    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc292f12-2b0a-4b5a-ac16-416e64c9040c-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:23.289145 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.289055    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7b7\" (UniqueName: \"kubernetes.io/projected/bc292f12-2b0a-4b5a-ac16-416e64c9040c-kube-api-access-cf7b7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:23.298329 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.298308    4546 scope.go:122] "RemoveContainer" containerID="da042cbf02064e7dd6e987e27263578e12d018ea683d298a1bf9c1f17b0dfbe8"
Nov 03 13:00:23.493183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.493154    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"]
Nov 03 13:00:23.497031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.497004    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-743d9b-29895180-jlnjz"]
Nov 03 13:00:23.597841 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.597730    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"]
Nov 03 13:00:23.602984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.602957    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895180-q78gt"]
Nov 03 13:00:23.700595 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.700564    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"]
Nov 03 13:00:23.703854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.703830    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895180-v25ql"]
Nov 03 13:00:23.858312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.858225    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"]
Nov 03 13:00:23.861564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:23.861537    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895180-8jwr4"]
Nov 03 13:00:24.139850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.139772    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 13:00:24.140206 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.140188    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 13:00:24.140922 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.140897    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 13:00:24.574320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.574290    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" path="/var/lib/kubelet/pods/1108ffa1-92ac-43f3-95c5-43d0fae9beed/volumes"
Nov 03 13:00:24.574768 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.574735    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" path="/var/lib/kubelet/pods/118b48b1-2a8b-4ec7-aad7-86d195fa586e/volumes"
Nov 03 13:00:24.575169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.575155    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201264ad-f94b-412c-ba91-24f933679a7e" path="/var/lib/kubelet/pods/201264ad-f94b-412c-ba91-24f933679a7e/volumes"
Nov 03 13:00:24.575887 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.575873    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" path="/var/lib/kubelet/pods/39e1e24d-d7ef-4b81-a640-ed4cd8733dcf/volumes"
Nov 03 13:00:24.576272 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.576259    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" path="/var/lib/kubelet/pods/4f55a1dd-f311-44ad-9c0c-fcee328c98a3/volumes"
Nov 03 13:00:24.576629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:24.576617    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" path="/var/lib/kubelet/pods/bc292f12-2b0a-4b5a-ac16-416e64c9040c/volumes"
Nov 03 13:00:27.567268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:27.567230    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:37.567043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:37.567006    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:37.653140 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:37.653062    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:00:39.231676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.231637    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"]
Nov 03 13:00:39.232261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232247    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232263    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232276    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232281    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232293    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232297    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232305    4546 cpu_manager.go:414] "containerMap: removing container" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:39.232307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232309    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232316    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232322    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232329    4546 cpu_manager.go:414] "containerMap: removing container" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232334    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232342    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232346    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232352    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232356    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232361    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232365    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232371    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232375    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232383    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232387    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232396    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232400    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232412    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d82fb07-7cb7-450c-ba71-7a533dc9603b" containerName="payload-pre-processing"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232417    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d82fb07-7cb7-450c-ba71-7a533dc9603b" containerName="payload-pre-processing"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232445    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232451    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232457    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232461    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232467    4546 cpu_manager.go:414] "containerMap: removing container" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232471    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232477    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232482    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232488    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232492    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232498    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232502    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232511    4546 cpu_manager.go:414] "containerMap: removing container" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232516    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232522    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232526    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232533    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.232522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232538    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232544    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232548    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232553    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232558    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232563    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232567    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232574    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232577    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232585    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6e71e1f9-e133-41ad-90a8-52bfffc343db" containerName="payload-pre-processing"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232589    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e71e1f9-e133-41ad-90a8-52bfffc343db" containerName="payload-pre-processing"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232595    4546 cpu_manager.go:414] "containerMap: removing container" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232598    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232605    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232609    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232616    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232620    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232626    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232630    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232636    4546 cpu_manager.go:414] "containerMap: removing container" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232640    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232646    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232650    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232656    4546 cpu_manager.go:414] "containerMap: removing container" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232660    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232743    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232768    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232773    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232779    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e71e1f9-e133-41ad-90a8-52bfffc343db" containerName="payload-pre-processing"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232785    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232790    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232797    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232803    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232808    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232813    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232819    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232825    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232829    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232836    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232841    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232847    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232852    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232858    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232862    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232867    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232873    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d82fb07-7cb7-450c-ba71-7a533dc9603b" containerName="payload-pre-processing"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232878    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232883    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232888    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232894    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232898    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232902    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232908    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232921    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f55a1dd-f311-44ad-9c0c-fcee328c98a3" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232926    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232932    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc292f12-2b0a-4b5a-ac16-416e64c9040c" containerName="cleanup"
Nov 03 13:00:39.233797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.232937    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:39.236639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.236617    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.297492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.297450    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.297702 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.297525    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4nlx\" (UniqueName: \"kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.297702 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.297644    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.305771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.305724    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:00:39.305927 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.305794    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:00:39.305927 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.305832    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:00:39.309167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.309143    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"]
Nov 03 13:00:39.332179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.332145    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:00:39.399470 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.399438    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.399641 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.399479    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4nlx\" (UniqueName: \"kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.399641 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.399528    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.400214 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.400193    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.402048 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.402015    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.452587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.452558    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4nlx\" (UniqueName: \"kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx\") pod \"maas-api-revoke-sub-a850b9f2-pwjh7\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.551417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.551329    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:39.918271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:39.918239    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"]
Nov 03 13:00:39.919590 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:39.919565    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f24dcba_d600_4f4c_b363_cf75a2fa0cc9.slice/crio-c0637b87269616b9f2d0a37fd2802d984411b15ed1a10db4baaf24e502764936 WatchSource:0}: Error finding container c0637b87269616b9f2d0a37fd2802d984411b15ed1a10db4baaf24e502764936: Status 404 returned error can't find the container with id c0637b87269616b9f2d0a37fd2802d984411b15ed1a10db4baaf24e502764936
Nov 03 13:00:40.389959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:40.389925    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7" event={"ID":"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9","Type":"ContainerStarted","Data":"bfcef184bcd86f810d16c018d0cd237f1f15fd894c54e6ebe70f3a1b06d1d390"}
Nov 03 13:00:40.390372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:40.389976    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7" event={"ID":"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9","Type":"ContainerStarted","Data":"c0637b87269616b9f2d0a37fd2802d984411b15ed1a10db4baaf24e502764936"}
Nov 03 13:00:40.498955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:40.498851    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7" podStartSLOduration=1.498835054 podStartE2EDuration="1.498835054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:00:40.495714769 +0000 UTC m=+4182.067298680" watchObservedRunningTime="2026-11-03 13:00:40.498835054 +0000 UTC m=+4182.070418964"
Nov 03 13:00:41.398205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:41.398174    4546 generic.go:356] "Generic (PLEG): container finished" podID="9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" containerID="bfcef184bcd86f810d16c018d0cd237f1f15fd894c54e6ebe70f3a1b06d1d390" exitCode=0
Nov 03 13:00:41.398602 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:41.398264    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7" event={"ID":"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9","Type":"ContainerDied","Data":"bfcef184bcd86f810d16c018d0cd237f1f15fd894c54e6ebe70f3a1b06d1d390"}
Nov 03 13:00:42.572879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.572851    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:42.741766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.741716    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx\" (UniqueName: \"kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx\") pod \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") "
Nov 03 13:00:42.741951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.741868    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca\") pod \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") "
Nov 03 13:00:42.742049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.742034    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca\") pod \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\" (UID: \"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9\") "
Nov 03 13:00:42.742246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.742223    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca" pod "9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" (UID: "9f24dcba-d600-4f4c-b363-cf75a2fa0cc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:42.742514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.742495    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:42.744105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.744081    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx" pod "9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" (UID: "9f24dcba-d600-4f4c-b363-cf75a2fa0cc9"). InnerVolumeSpecName "kube-api-access-m4nlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:42.744219 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.744121    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca" pod "9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" (UID: "9f24dcba-d600-4f4c-b363-cf75a2fa0cc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:00:42.843527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.843503    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:42.843527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:42.843524    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4nlx\" (UniqueName: \"kubernetes.io/projected/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9-kube-api-access-m4nlx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:43.414066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:43.414037    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"
Nov 03 13:00:43.414247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:43.414067    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7" event={"ID":"9f24dcba-d600-4f4c-b363-cf75a2fa0cc9","Type":"ContainerDied","Data":"c0637b87269616b9f2d0a37fd2802d984411b15ed1a10db4baaf24e502764936"}
Nov 03 13:00:43.414247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:43.414120    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0637b87269616b9f2d0a37fd2802d984411b15ed1a10db4baaf24e502764936"
Nov 03 13:00:47.567146 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:47.567104    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:50.602276 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602242    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"]
Nov 03 13:00:50.602895 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602879    4546 cpu_manager.go:414] "containerMap: removing container" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602895    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602912    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602917    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602927    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602932    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602944    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602949    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d847af6-74a2-45cb-9a6b-74750275610e" containerName="cleanup"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602960    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" containerName="revoke-keys"
Nov 03 13:00:50.602966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602964    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" containerName="revoke-keys"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.602996    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603000    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603009    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603013    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1108ffa1-92ac-43f3-95c5-43d0fae9beed" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603021    4546 cpu_manager.go:414] "containerMap: removing container" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603027    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603117    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e1e24d-d7ef-4b81-a640-ed4cd8733dcf" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603129    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="118b48b1-2a8b-4ec7-aad7-86d195fa586e" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603137    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="201264ad-f94b-412c-ba91-24f933679a7e" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603142    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11e4db1-4807-411e-a3dc-cf95d30ddd22" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603150    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" containerName="revoke-keys"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603158    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad320d7-8fd6-421c-afde-25c47719c2d9" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603166    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc7faed-4d52-4e8d-aed3-149fd75522a6" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603172    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aaace4b-7c89-47ad-be3e-9582d747f3dd" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603176    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9aa90ac-39fd-4fa8-8f45-bba34671884f" containerName="cleanup"
Nov 03 13:00:50.603248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.603182    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c496b387-af0f-4d62-a5ca-e0029eea7f5f" containerName="cleanup"
Nov 03 13:00:50.606957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.606931    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.646843 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.646817    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.647042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.647021    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64w8l\" (UniqueName: \"kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.647195 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.647170    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.660381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.660358    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"]
Nov 03 13:00:50.702064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.702018    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:00:50.702064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.702064    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:00:50.702274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.702097    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:00:50.702274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.702111    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:00:50.748572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.748540    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.748732 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.748662    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64w8l\" (UniqueName: \"kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.748803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.748784    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.749253 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.749235    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.751449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.751424    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.802298 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.802264    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64w8l\" (UniqueName: \"kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l\") pod \"maas-api-revoke-sub-ff4be8bc-6fvk5\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:50.923133 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:50.923057    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:51.295059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:51.295025    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"]
Nov 03 13:00:51.295787 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:00:51.295732    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64c2197_92ca_4ad3_a80b_4383f3c26c5f.slice/crio-7c739af0ff33c4213af9c16d003fdc5da80c8c4207a1043a766d59d4cb55d6d8 WatchSource:0}: Error finding container 7c739af0ff33c4213af9c16d003fdc5da80c8c4207a1043a766d59d4cb55d6d8: Status 404 returned error can't find the container with id 7c739af0ff33c4213af9c16d003fdc5da80c8c4207a1043a766d59d4cb55d6d8
Nov 03 13:00:51.477781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:51.477730    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5" event={"ID":"e64c2197-92ca-4ad3-a80b-4383f3c26c5f","Type":"ContainerStarted","Data":"7c739af0ff33c4213af9c16d003fdc5da80c8c4207a1043a766d59d4cb55d6d8"}
Nov 03 13:00:51.594238 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:51.594164    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"]
Nov 03 13:00:52.486375 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:52.486344    4546 generic.go:356] "Generic (PLEG): container finished" podID="e64c2197-92ca-4ad3-a80b-4383f3c26c5f" containerID="4c460b7d74a447228d2154c5d6167cbd21a23c29e1263fe9d7f875fa26f3ba6b" exitCode=0
Nov 03 13:00:52.486863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:52.486402    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5" event={"ID":"e64c2197-92ca-4ad3-a80b-4383f3c26c5f","Type":"ContainerDied","Data":"4c460b7d74a447228d2154c5d6167cbd21a23c29e1263fe9d7f875fa26f3ba6b"}
Nov 03 13:00:53.648899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.648872    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:53.712673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.712648    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca\") pod \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") "
Nov 03 13:00:53.712871 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.712788    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l\" (UniqueName: \"kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l\") pod \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") "
Nov 03 13:00:53.712959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.712901    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca\") pod \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\" (UID: \"e64c2197-92ca-4ad3-a80b-4383f3c26c5f\") "
Nov 03 13:00:53.713247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.713225    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca" pod "e64c2197-92ca-4ad3-a80b-4383f3c26c5f" (UID: "e64c2197-92ca-4ad3-a80b-4383f3c26c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:53.713693 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.713672    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:53.715041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.715014    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca" pod "e64c2197-92ca-4ad3-a80b-4383f3c26c5f" (UID: "e64c2197-92ca-4ad3-a80b-4383f3c26c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:00:53.715130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.715076    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l" pod "e64c2197-92ca-4ad3-a80b-4383f3c26c5f" (UID: "e64c2197-92ca-4ad3-a80b-4383f3c26c5f"). InnerVolumeSpecName "kube-api-access-64w8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:53.814580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.814502    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:53.814580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:53.814533    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64w8l\" (UniqueName: \"kubernetes.io/projected/e64c2197-92ca-4ad3-a80b-4383f3c26c5f-kube-api-access-64w8l\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:54.502225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:54.502199    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"
Nov 03 13:00:54.502381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:54.502242    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5" event={"ID":"e64c2197-92ca-4ad3-a80b-4383f3c26c5f","Type":"ContainerDied","Data":"7c739af0ff33c4213af9c16d003fdc5da80c8c4207a1043a766d59d4cb55d6d8"}
Nov 03 13:00:54.502381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:54.502293    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c739af0ff33c4213af9c16d003fdc5da80c8c4207a1043a766d59d4cb55d6d8"
Nov 03 13:00:56.594451 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.594412    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh" podUID="c9bf56ec-20f3-45c0-ade4-80192e2c7c21" containerName="payload-processing" containerID="cri-o://88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9" gracePeriod=25
Nov 03 13:00:56.863522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.863455    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 13:00:56.954398 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.954358    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume\") pod \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") "
Nov 03 13:00:56.954578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.954540    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86\" (UniqueName: \"kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86\") pod \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\" (UID: \"c9bf56ec-20f3-45c0-ade4-80192e2c7c21\") "
Nov 03 13:00:56.954727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.954706    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume" pod "c9bf56ec-20f3-45c0-ade4-80192e2c7c21" (UID: "c9bf56ec-20f3-45c0-ade4-80192e2c7c21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:00:56.954990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.954964    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:56.956823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:56.956803    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86" pod "c9bf56ec-20f3-45c0-ade4-80192e2c7c21" (UID: "c9bf56ec-20f3-45c0-ade4-80192e2c7c21"). InnerVolumeSpecName "kube-api-access-7mt86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:00:57.056454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.056408    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mt86\" (UniqueName: \"kubernetes.io/projected/c9bf56ec-20f3-45c0-ade4-80192e2c7c21-kube-api-access-7mt86\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:00:57.524978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.524947    4546 generic.go:356] "Generic (PLEG): container finished" podID="c9bf56ec-20f3-45c0-ade4-80192e2c7c21" containerID="88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9" exitCode=0
Nov 03 13:00:57.525212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.525007    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"
Nov 03 13:00:57.525212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.525040    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh" event={"ID":"c9bf56ec-20f3-45c0-ade4-80192e2c7c21","Type":"ContainerDied","Data":"88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9"}
Nov 03 13:00:57.525212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.525097    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh" event={"ID":"c9bf56ec-20f3-45c0-ade4-80192e2c7c21","Type":"ContainerDied","Data":"14b99b96dae4e6f88197de398c53eddaf5125053653d8612e89e23be22deebf2"}
Nov 03 13:00:57.525212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.525123    4546 scope.go:122] "RemoveContainer" containerID="88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9"
Nov 03 13:00:57.536004 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.535987    4546 scope.go:122] "RemoveContainer" containerID="88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9"
Nov 03 13:00:57.536292 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:00:57.536274    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9\": container with ID starting with 88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9 not found: ID does not exist" containerID="88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9"
Nov 03 13:00:57.536364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.536299    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9"} err="failed to get container status \"88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9\": rpc error: code = NotFound desc = could not find container \"88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9\": container with ID starting with 88a207753a87546df4e3311133d1f3cfbbe09add80ae6c64ff171337be90bfd9 not found: ID does not exist"
Nov 03 13:00:57.567234 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.567194    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:00:57.677948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.677902    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"]
Nov 03 13:00:57.682979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:57.682951    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d126a6-776bd84c77-gp7bh"]
Nov 03 13:00:58.576223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:00:58.576187    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bf56ec-20f3-45c0-ade4-80192e2c7c21" path="/var/lib/kubelet/pods/c9bf56ec-20f3-45c0-ade4-80192e2c7c21/volumes"
Nov 03 13:01:01.863605 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:01.863574    4546 scope.go:122] "RemoveContainer" containerID="7e3b4cca12fd9a203433fc53ce92732f39e06136ec54726a4fabc2a6413d12ba"
Nov 03 13:01:04.546790 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.546706    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"]
Nov 03 13:01:04.547601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.547579    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c9bf56ec-20f3-45c0-ade4-80192e2c7c21" containerName="payload-processing"
Nov 03 13:01:04.547713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.547603    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bf56ec-20f3-45c0-ade4-80192e2c7c21" containerName="payload-processing"
Nov 03 13:01:04.547713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.547625    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e64c2197-92ca-4ad3-a80b-4383f3c26c5f" containerName="revoke-keys"
Nov 03 13:01:04.547713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.547631    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64c2197-92ca-4ad3-a80b-4383f3c26c5f" containerName="revoke-keys"
Nov 03 13:01:04.547909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.547856    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bf56ec-20f3-45c0-ade4-80192e2c7c21" containerName="payload-processing"
Nov 03 13:01:04.547909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.547875    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64c2197-92ca-4ad3-a80b-4383f3c26c5f" containerName="revoke-keys"
Nov 03 13:01:04.551426 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.551406    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.564392 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.564365    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.564512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.564499    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.564590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.564579    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jwq\" (UniqueName: \"kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.600462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.600440    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:01:04.604432 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.604409    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"]
Nov 03 13:01:04.646276 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.646233    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:01:04.646472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.646286    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:01:04.646472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.646294    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:01:04.665715 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.665685    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.665864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.665815    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.665937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.665921    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jwq\" (UniqueName: \"kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.666412 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.666394    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.668376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.668358    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.720405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.720379    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jwq\" (UniqueName: \"kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq\") pod \"maas-api-revoke-sub-764464d0-j489n\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:04.865228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:04.865145    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:05.237844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:05.237812    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"]
Nov 03 13:01:05.238970 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:01:05.238938    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083f6dcf_8c1d_419f_a30d_08c0892f1274.slice/crio-1738581e9e1c3f2011a483ec89f7dc582c73aad11ba7be64bd2e0f5ada12d377 WatchSource:0}: Error finding container 1738581e9e1c3f2011a483ec89f7dc582c73aad11ba7be64bd2e0f5ada12d377: Status 404 returned error can't find the container with id 1738581e9e1c3f2011a483ec89f7dc582c73aad11ba7be64bd2e0f5ada12d377
Nov 03 13:01:05.580169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:05.580138    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n" event={"ID":"083f6dcf-8c1d-419f-a30d-08c0892f1274","Type":"ContainerStarted","Data":"2601901dc25dc03e179d9e2fe6c433a0858729c20812ff2e6933813abcdac916"}
Nov 03 13:01:05.580576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:05.580187    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n" event={"ID":"083f6dcf-8c1d-419f-a30d-08c0892f1274","Type":"ContainerStarted","Data":"1738581e9e1c3f2011a483ec89f7dc582c73aad11ba7be64bd2e0f5ada12d377"}
Nov 03 13:01:05.690056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:05.690007    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n" podStartSLOduration=1.689994354 podStartE2EDuration="1.689994354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:01:05.687211956 +0000 UTC m=+4207.258795871" watchObservedRunningTime="2026-11-03 13:01:05.689994354 +0000 UTC m=+4207.261578287"
Nov 03 13:01:06.587994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:06.587958    4546 generic.go:356] "Generic (PLEG): container finished" podID="083f6dcf-8c1d-419f-a30d-08c0892f1274" containerID="2601901dc25dc03e179d9e2fe6c433a0858729c20812ff2e6933813abcdac916" exitCode=0
Nov 03 13:01:06.588427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:06.588036    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n" event={"ID":"083f6dcf-8c1d-419f-a30d-08c0892f1274","Type":"ContainerDied","Data":"2601901dc25dc03e179d9e2fe6c433a0858729c20812ff2e6933813abcdac916"}
Nov 03 13:01:07.567256 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.567219    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:07.653429 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:01:07.653395    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:01:07.764494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.764472    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:07.806831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.806805    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca\") pod \"083f6dcf-8c1d-419f-a30d-08c0892f1274\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") "
Nov 03 13:01:07.806983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.806918    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca\") pod \"083f6dcf-8c1d-419f-a30d-08c0892f1274\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") "
Nov 03 13:01:07.807032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.807001    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq\" (UniqueName: \"kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq\") pod \"083f6dcf-8c1d-419f-a30d-08c0892f1274\" (UID: \"083f6dcf-8c1d-419f-a30d-08c0892f1274\") "
Nov 03 13:01:07.807256 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.807232    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca" pod "083f6dcf-8c1d-419f-a30d-08c0892f1274" (UID: "083f6dcf-8c1d-419f-a30d-08c0892f1274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:07.807434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.807419    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:07.809044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.809021    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca" pod "083f6dcf-8c1d-419f-a30d-08c0892f1274" (UID: "083f6dcf-8c1d-419f-a30d-08c0892f1274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:07.809160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.809143    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq" pod "083f6dcf-8c1d-419f-a30d-08c0892f1274" (UID: "083f6dcf-8c1d-419f-a30d-08c0892f1274"). InnerVolumeSpecName "kube-api-access-j6jwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:07.908833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.908795    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6jwq\" (UniqueName: \"kubernetes.io/projected/083f6dcf-8c1d-419f-a30d-08c0892f1274-kube-api-access-j6jwq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:07.908833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:07.908838    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083f6dcf-8c1d-419f-a30d-08c0892f1274-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:08.604195 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:08.604166    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"
Nov 03 13:01:08.604382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:08.604205    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n" event={"ID":"083f6dcf-8c1d-419f-a30d-08c0892f1274","Type":"ContainerDied","Data":"1738581e9e1c3f2011a483ec89f7dc582c73aad11ba7be64bd2e0f5ada12d377"}
Nov 03 13:01:08.604382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:08.604249    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1738581e9e1c3f2011a483ec89f7dc582c73aad11ba7be64bd2e0f5ada12d377"
Nov 03 13:01:17.576876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:17.576814    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:27.567459 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:27.567421    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:34.688656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:34.688624    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"]
Nov 03 13:01:34.689133 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:34.688855    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" containerID="cri-o://972ff6d949239b86379984b9ce25369f5b712a548195d2e321f8c3fd08f141f4" gracePeriod=30
Nov 03 13:01:35.046947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:35.046895    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"]
Nov 03 13:01:35.050386 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:35.050359    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7fd9281-rkrg4"]
Nov 03 13:01:36.575151 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:36.575113    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f" path="/var/lib/kubelet/pods/ae8b4d25-8fb8-4c21-8b3f-35ef63a5918f/volumes"
Nov 03 13:01:37.567494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:37.567454    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:37.654632 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:01:37.654596    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:01:39.142887 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:39.142856    4546 patch_prober.go:29] interesting pod/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:01:39.143273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:39.142924    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:01:39.911843 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:39.911436    4546 generic.go:356] "Generic (PLEG): container finished" podID="2cc20d22-0d86-45f9-af45-6af554757811" containerID="972ff6d949239b86379984b9ce25369f5b712a548195d2e321f8c3fd08f141f4" exitCode=0
Nov 03 13:01:39.911843 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:39.911614    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" event={"ID":"2cc20d22-0d86-45f9-af45-6af554757811","Type":"ContainerDied","Data":"972ff6d949239b86379984b9ce25369f5b712a548195d2e321f8c3fd08f141f4"}
Nov 03 13:01:39.911843 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:39.911703    4546 scope.go:122] "RemoveContainer" containerID="b5167c42a5967110250e8e6d867d0ca6ca757db51f436797068051db9d32de98"
Nov 03 13:01:39.974703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:39.974678    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 13:01:40.054428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054398    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054467    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054499    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054557    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054601    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054619    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054679    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054793    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054827    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token\") pod \"2cc20d22-0d86-45f9-af45-6af554757811\" (UID: \"2cc20d22-0d86-45f9-af45-6af554757811\") "
Nov 03 13:01:40.054983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054844    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:40.055215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.054987    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:40.055285 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055234    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:40.055285 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055266    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:40.055645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055529    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:40.055645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055587    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.055645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055614    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc20d22-0d86-45f9-af45-6af554757811-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.055645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055628    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.055645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.055642    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.057222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.057198    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "kube-api-access-hlz2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:40.057337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.057316    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:01:40.057337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.057317    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:01:40.057459 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.057443    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token" pod "2cc20d22-0d86-45f9-af45-6af554757811" (UID: "2cc20d22-0d86-45f9-af45-6af554757811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:40.156991 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.156960    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc20d22-0d86-45f9-af45-6af554757811-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.156991 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.156988    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.157420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.157000    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc20d22-0d86-45f9-af45-6af554757811-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.157420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.157013    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlz2w\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-kube-api-access-hlz2w\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.157420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.157024    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc20d22-0d86-45f9-af45-6af554757811-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:40.919562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.919482    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"
Nov 03 13:01:40.919562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.919516    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv" event={"ID":"2cc20d22-0d86-45f9-af45-6af554757811","Type":"ContainerDied","Data":"8f07c5c4e636580632ed2deb768c8ad549ef33a141b92dbf613a097f7cd97412"}
Nov 03 13:01:40.919807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:40.919573    4546 scope.go:122] "RemoveContainer" containerID="972ff6d949239b86379984b9ce25369f5b712a548195d2e321f8c3fd08f141f4"
Nov 03 13:01:41.120493 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:41.120462    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"]
Nov 03 13:01:41.165375 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:41.165345    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb74b5-openshift-default-6599f86bc7-2hvpv"]
Nov 03 13:01:42.575078 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:42.575042    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc20d22-0d86-45f9-af45-6af554757811" path="/var/lib/kubelet/pods/2cc20d22-0d86-45f9-af45-6af554757811/volumes"
Nov 03 13:01:43.745682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.745646    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"]
Nov 03 13:01:43.746473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746450    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy"
Nov 03 13:01:43.746573 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746474    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy"
Nov 03 13:01:43.746573 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746553    4546 cpu_manager.go:414] "containerMap: removing container" podUID="083f6dcf-8c1d-419f-a30d-08c0892f1274" containerName="revoke-keys"
Nov 03 13:01:43.746573 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746563    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083f6dcf-8c1d-419f-a30d-08c0892f1274" containerName="revoke-keys"
Nov 03 13:01:43.746737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746695    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy"
Nov 03 13:01:43.746737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746711    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="083f6dcf-8c1d-419f-a30d-08c0892f1274" containerName="revoke-keys"
Nov 03 13:01:43.746737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.746723    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy"
Nov 03 13:01:43.751802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.751782    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:43.800774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.800716    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:01:43.803640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.803610    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"]
Nov 03 13:01:43.845394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.845354    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:01:43.884051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.884019    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:01:43.884448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.884430    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:01:43.921808 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.916697    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:43.940822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.940699    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:43.941054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:43.940966    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.042295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.042209    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.042295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.042262    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.042504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.042321    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.042892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.042866    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.045100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.045073    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.096316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.096286    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x\") pod \"maas-api-revoke-sub-c62eb92a-lmq8z\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.367014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.366925    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:44.768836 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.753818    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"]
Nov 03 13:01:44.974319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:44.973995    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z" event={"ID":"20878294-56a9-44e3-826b-0bc2aee496ab","Type":"ContainerStarted","Data":"03a9ba325e7c00549c6f5bb2ebc53658aaf4fe9a2e52e69242e02199a4b6d875"}
Nov 03 13:01:46.001780 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:45.998145    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z" event={"ID":"20878294-56a9-44e3-826b-0bc2aee496ab","Type":"ContainerStarted","Data":"f74ff5dc639c2793c4f50b80a84e07a3b7efc2a16dffad37fe13a7f70ccabe2e"}
Nov 03 13:01:47.105084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.104394    4546 generic.go:356] "Generic (PLEG): container finished" podID="20878294-56a9-44e3-826b-0bc2aee496ab" containerID="f74ff5dc639c2793c4f50b80a84e07a3b7efc2a16dffad37fe13a7f70ccabe2e" exitCode=0
Nov 03 13:01:47.105084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.104523    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z" event={"ID":"20878294-56a9-44e3-826b-0bc2aee496ab","Type":"ContainerDied","Data":"f74ff5dc639c2793c4f50b80a84e07a3b7efc2a16dffad37fe13a7f70ccabe2e"}
Nov 03 13:01:47.211883 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.211843    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:01:47.212188 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.212157    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:47.526777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.521967    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 13:01:47.589809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.589022    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:47.628777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.615844    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:47.628777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.616703    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:01:47.687787 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.687225    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:47.713294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.712860    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:47.713939 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.713580    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:47.801622 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:47.801519    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:01:48.268146 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.268117    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:48.387704 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.387621    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca\") pod \"20878294-56a9-44e3-826b-0bc2aee496ab\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") "
Nov 03 13:01:48.387926 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.387816    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x\") pod \"20878294-56a9-44e3-826b-0bc2aee496ab\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") "
Nov 03 13:01:48.387926 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.387885    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca\") pod \"20878294-56a9-44e3-826b-0bc2aee496ab\" (UID: \"20878294-56a9-44e3-826b-0bc2aee496ab\") "
Nov 03 13:01:48.388266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.388238    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca" pod "20878294-56a9-44e3-826b-0bc2aee496ab" (UID: "20878294-56a9-44e3-826b-0bc2aee496ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:48.388546 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.388524    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:48.390405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.390371    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca" pod "20878294-56a9-44e3-826b-0bc2aee496ab" (UID: "20878294-56a9-44e3-826b-0bc2aee496ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:48.390559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.390543    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x" pod "20878294-56a9-44e3-826b-0bc2aee496ab" (UID: "20878294-56a9-44e3-826b-0bc2aee496ab"). InnerVolumeSpecName "kube-api-access-zrb9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:48.489632 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.489603    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20878294-56a9-44e3-826b-0bc2aee496ab-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:48.489632 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:48.489631    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/20878294-56a9-44e3-826b-0bc2aee496ab-kube-api-access-zrb9x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:49.119628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:49.119599    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"
Nov 03 13:01:49.119833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:49.119644    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z" event={"ID":"20878294-56a9-44e3-826b-0bc2aee496ab","Type":"ContainerDied","Data":"03a9ba325e7c00549c6f5bb2ebc53658aaf4fe9a2e52e69242e02199a4b6d875"}
Nov 03 13:01:49.119833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:49.119690    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03a9ba325e7c00549c6f5bb2ebc53658aaf4fe9a2e52e69242e02199a4b6d875"
Nov 03 13:01:56.256598 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:56.256564    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"]
Nov 03 13:01:56.258102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:01:56.258069    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:02:01.257845 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.257786    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing" containerID="cri-o://c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28" gracePeriod=25
Nov 03 13:02:01.550267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.550241    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 13:02:01.716385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.716342    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d\") pod \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") "
Nov 03 13:02:01.716646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.716627    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume\") pod \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") "
Nov 03 13:02:01.717078 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.717046    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume\") pod \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") "
Nov 03 13:02:01.717224 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.717148    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials\") pod \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\" (UID: \"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43\") "
Nov 03 13:02:01.718817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.718786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume" pod "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" (UID: "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:01.719242 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.719217    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume" pod "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" (UID: "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:01.732031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.731957    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d" pod "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" (UID: "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43"). InnerVolumeSpecName "kube-api-access-hq66d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:01.732031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.732005    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials" pod "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" (UID: "e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:01.819669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.819582    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:01.819669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.819611    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:01.819669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.819620    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq66d\" (UniqueName: \"kubernetes.io/projected/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-kube-api-access-hq66d\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:01.819669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:01.819630    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:02.229294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.229244    4546 generic.go:356] "Generic (PLEG): container finished" podID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerID="c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28" exitCode=0
Nov 03 13:02:02.229587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.229566    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" event={"ID":"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43","Type":"ContainerDied","Data":"c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28"}
Nov 03 13:02:02.229719 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.229703    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9" event={"ID":"e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43","Type":"ContainerDied","Data":"99eca470794ff5d29d9257a94f4a4945a8acd9ce5c42bb8cbbc28370840ca78e"}
Nov 03 13:02:02.229853 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.229742    4546 scope.go:122] "RemoveContainer" containerID="c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28"
Nov 03 13:02:02.230091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.230063    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"
Nov 03 13:02:02.244335 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.244308    4546 scope.go:122] "RemoveContainer" containerID="c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28"
Nov 03 13:02:02.244681 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:02:02.244654    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28\": container with ID starting with c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28 not found: ID does not exist" containerID="c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28"
Nov 03 13:02:02.244793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.244688    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28"} err="failed to get container status \"c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28\": rpc error: code = NotFound desc = could not find container \"c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28\": container with ID starting with c92735163379ed7191c3d9662061e964f1ce8803a546079be6d50de1121a4e28 not found: ID does not exist"
Nov 03 13:02:02.385313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.385283    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"]
Nov 03 13:02:02.432114 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.432078    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b96957bd6-zd8h9"]
Nov 03 13:02:02.574783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:02.574691    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" path="/var/lib/kubelet/pods/e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43/volumes"
Nov 03 13:02:03.574023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:03.573990    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"]
Nov 03 13:02:03.574505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:03.574225    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" containerID="cri-o://292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618" gracePeriod=30
Nov 03 13:02:07.463387 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:07.463349    4546 patch_prober.go:29] interesting pod/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:02:07.463798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:07.463410    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:02:07.653038 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:02:07.652965    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:02:08.839193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.839128    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 13:02:08.955141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955114    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955150    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955183    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955300    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955409    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:02:08.955515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955417    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955504    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:02:08.955682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955527    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955570    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955635    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.955682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955678    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket\") pod \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\" (UID: \"932c34dc-3919-4ec7-bce3-50d81bdd1ddb\") "
Nov 03 13:02:08.956105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.955823    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:02:08.956105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.956087    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:02:08.956242 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.956222    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:08.956242 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.956236    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:08.956364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.956251    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:08.956364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.956264    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:08.956364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.956230    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:08.957587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.957564    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:02:08.957689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.957674    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "kube-api-access-8jjml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:08.958134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.958111    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:08.958216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:08.958118    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo" pod "932c34dc-3919-4ec7-bce3-50d81bdd1ddb" (UID: "932c34dc-3919-4ec7-bce3-50d81bdd1ddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:02:09.057320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.057288    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:09.057320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.057315    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:09.057320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.057326    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:09.057576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.057356    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jjml\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-kube-api-access-8jjml\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:09.057576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.057367    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932c34dc-3919-4ec7-bce3-50d81bdd1ddb-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:09.280816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.280784    4546 generic.go:356] "Generic (PLEG): container finished" podID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerID="292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618" exitCode=0
Nov 03 13:02:09.280997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.280841    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"
Nov 03 13:02:09.280997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.280841    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" event={"ID":"932c34dc-3919-4ec7-bce3-50d81bdd1ddb","Type":"ContainerDied","Data":"292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618"}
Nov 03 13:02:09.281106 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.280995    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6" event={"ID":"932c34dc-3919-4ec7-bce3-50d81bdd1ddb","Type":"ContainerDied","Data":"eec256e6c4a63267c3d85cdf45609beca8a7ea9c3559e141906a08058a517b80"}
Nov 03 13:02:09.281106 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.281022    4546 scope.go:122] "RemoveContainer" containerID="292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618"
Nov 03 13:02:09.292454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.292436    4546 scope.go:122] "RemoveContainer" containerID="0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"
Nov 03 13:02:09.301472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.301453    4546 scope.go:122] "RemoveContainer" containerID="292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618"
Nov 03 13:02:09.301715 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:02:09.301697    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618\": container with ID starting with 292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618 not found: ID does not exist" containerID="292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618"
Nov 03 13:02:09.301789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.301724    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618"} err="failed to get container status \"292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618\": rpc error: code = NotFound desc = could not find container \"292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618\": container with ID starting with 292fb57762e1d22fe27afb73a01bc78baef67f2ce10737f1fb4ca15f5b690618 not found: ID does not exist"
Nov 03 13:02:09.301789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.301742    4546 scope.go:122] "RemoveContainer" containerID="0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"
Nov 03 13:02:09.301998 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:02:09.301984    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2\": container with ID starting with 0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2 not found: ID does not exist" containerID="0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"
Nov 03 13:02:09.302081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.302001    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2"} err="failed to get container status \"0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2\": rpc error: code = NotFound desc = could not find container \"0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2\": container with ID starting with 0f82db4d9732edf5863be5d48976596e25999adce651056b71e40708a43a89b2 not found: ID does not exist"
Nov 03 13:02:09.435792 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.435760    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"]
Nov 03 13:02:09.481822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:09.481796    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d126a6-openshift-default-698d5ccf99-kt4m6"]
Nov 03 13:02:10.574571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:10.574540    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" path="/var/lib/kubelet/pods/932c34dc-3919-4ec7-bce3-50d81bdd1ddb/volumes"
Nov 03 13:02:19.888759 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.888715    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"]
Nov 03 13:02:19.889349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889334    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889350    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889359    4546 cpu_manager.go:414] "containerMap: removing container" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889364    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889373    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889377    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc20d22-0d86-45f9-af45-6af554757811" containerName="istio-proxy"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889393    4546 cpu_manager.go:414] "containerMap: removing container" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889399    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889410    4546 cpu_manager.go:414] "containerMap: removing container" podUID="20878294-56a9-44e3-826b-0bc2aee496ab" containerName="revoke-keys"
Nov 03 13:02:19.889421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889414    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20878294-56a9-44e3-826b-0bc2aee496ab" containerName="revoke-keys"
Nov 03 13:02:19.889786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889509    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="20878294-56a9-44e3-826b-0bc2aee496ab" containerName="revoke-keys"
Nov 03 13:02:19.889786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889523    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e539c7bc-493c-4b01-a7ad-4f3ab3dc4e43" containerName="payload-processing"
Nov 03 13:02:19.889786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889532    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy"
Nov 03 13:02:19.889786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.889540    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="932c34dc-3919-4ec7-bce3-50d81bdd1ddb" containerName="istio-proxy"
Nov 03 13:02:19.894323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.894307    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:19.992371 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:19.992333    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"]
Nov 03 13:02:20.045165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.043654    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6bd346ef-openshift-default-dockercfg-p6zrb\""
Nov 03 13:02:20.046491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.046439    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.046828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.046807    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.047131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.047105    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.047457 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.047439    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.047965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.047945    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.048254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.048213    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.048501 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.048440    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.048845 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.048687    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.049064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.049043    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgv5\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.151790    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.151867    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.151894    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.151945    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.155760    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgv5\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.155870    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.155919    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.156097    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.156647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.156205    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.157267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.156685    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.157267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.157214    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.157381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.157329    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.157527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.157491    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.157671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.157643    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.160264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.160223    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.167427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.167405    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.209718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.209684    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgv5\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.210545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.210520    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token\") pod \"e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") " pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.508371 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.508345    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:20.930525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.930495    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"]
Nov 03 13:02:20.931207 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:02:20.931181    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8e7a74_be1b_4be1_9a28_330448552921.slice/crio-6d69dd6229f8b7971ebef5164f2899ea09ce0abd1f9723d39872db5975b50573 WatchSource:0}: Error finding container 6d69dd6229f8b7971ebef5164f2899ea09ce0abd1f9723d39872db5975b50573: Status 404 returned error can't find the container with id 6d69dd6229f8b7971ebef5164f2899ea09ce0abd1f9723d39872db5975b50573
Nov 03 13:02:20.933883 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.933855    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:02:20.933946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.933916    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:02:20.933984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:20.933946    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:02:21.384087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:21.384056    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" event={"ID":"fe8e7a74-be1b-4be1-9a28-330448552921","Type":"ContainerStarted","Data":"2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"}
Nov 03 13:02:21.384279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:21.384112    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" event={"ID":"fe8e7a74-be1b-4be1-9a28-330448552921","Type":"ContainerStarted","Data":"6d69dd6229f8b7971ebef5164f2899ea09ce0abd1f9723d39872db5975b50573"}
Nov 03 13:02:21.509017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:21.508979    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:21.514936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:21.514907    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:21.585194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:21.585154    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podStartSLOduration=2.585141194 podStartE2EDuration="2.585141194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:21.538506953 +0000 UTC m=+4283.110090863" watchObservedRunningTime="2026-11-03 13:02:21.585141194 +0000 UTC m=+4283.156725108"
Nov 03 13:02:22.390816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:22.390784    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:22.391731 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:22.391709    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:02:37.516397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:37.516343    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:02:37.630892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:37.630833    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:02:37.741775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:37.740875    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:02:37.745867 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:02:37.745834    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:02:37.811005 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:02:37.809102    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:03:01.981647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:01.981612    4546 scope.go:122] "RemoveContainer" containerID="37ababcb573adf8284627224a6719ab666625f182164c91914e0973f05790928"
Nov 03 13:03:07.652224 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:03:07.652189    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:03:33.216152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:33.216115    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"]
Nov 03 13:03:33.216797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:33.216340    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy" containerID="cri-o://1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd" gracePeriod=30
Nov 03 13:03:37.651878 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:03:37.651785    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:03:38.505477 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.505452    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 13:03:38.637953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.637916    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.637953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.637961    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.637992    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638019    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638056    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638075    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638149    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638244    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:03:38.638562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638274    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638331    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data\") pod \"ed666c5a-f22a-44f2-842c-b350cf68825e\" (UID: \"ed666c5a-f22a-44f2-842c-b350cf68825e\") "
Nov 03 13:03:38.638562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638370    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:03:38.638883 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638564    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:03:38.638883 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638469    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:03:38.638883 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638666    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:03:38.639050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638968    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.639050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.638990    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.639050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.639003    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.639050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.639016    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed666c5a-f22a-44f2-842c-b350cf68825e-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.639050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.639030    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.640920 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.640886    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:03:38.641064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.640950    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "kube-api-access-4tqpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:03:38.641212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.641193    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:03:38.641212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.641204    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy" pod "ed666c5a-f22a-44f2-842c-b350cf68825e" (UID: "ed666c5a-f22a-44f2-842c-b350cf68825e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:03:38.740222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.740130    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tqpr\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-kube-api-access-4tqpr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.740222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.740166    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.740222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.740177    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.740222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.740186    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed666c5a-f22a-44f2-842c-b350cf68825e-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:38.948136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.948105    4546 generic.go:356] "Generic (PLEG): container finished" podID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerID="1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd" exitCode=0
Nov 03 13:03:38.948345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.948159    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"
Nov 03 13:03:38.948345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.948188    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" event={"ID":"ed666c5a-f22a-44f2-842c-b350cf68825e","Type":"ContainerDied","Data":"1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd"}
Nov 03 13:03:38.948345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.948242    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2" event={"ID":"ed666c5a-f22a-44f2-842c-b350cf68825e","Type":"ContainerDied","Data":"412605148ff2be6bd79e8208da7301aec3d5c41b17908bd09eae1b33c901beeb"}
Nov 03 13:03:38.948345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.948265    4546 scope.go:122] "RemoveContainer" containerID="1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd"
Nov 03 13:03:38.963611 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.963588    4546 scope.go:122] "RemoveContainer" containerID="52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"
Nov 03 13:03:38.973502 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.973480    4546 scope.go:122] "RemoveContainer" containerID="1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd"
Nov 03 13:03:38.973911 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:03:38.973890    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd\": container with ID starting with 1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd not found: ID does not exist" containerID="1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd"
Nov 03 13:03:38.973975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.973923    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd"} err="failed to get container status \"1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd\": rpc error: code = NotFound desc = could not find container \"1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd\": container with ID starting with 1be10dfecfabe00b7c4d791c6c247938832a9f8384519e4381d365e4f860cddd not found: ID does not exist"
Nov 03 13:03:38.973975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.973941    4546 scope.go:122] "RemoveContainer" containerID="52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"
Nov 03 13:03:38.974297 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:03:38.974279    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2\": container with ID starting with 52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2 not found: ID does not exist" containerID="52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"
Nov 03 13:03:38.974355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:38.974302    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2"} err="failed to get container status \"52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2\": rpc error: code = NotFound desc = could not find container \"52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2\": container with ID starting with 52baa90e95585793447711f13ee00f1a01f68ac42b1c5b24b69616267aadd5b2 not found: ID does not exist"
Nov 03 13:03:39.104094 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:39.104005    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"]
Nov 03 13:03:39.150417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:39.150377    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-aa9221-openshift-default-7d66458957-vgtj2"]
Nov 03 13:03:40.573932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:03:40.573900    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" path="/var/lib/kubelet/pods/ed666c5a-f22a-44f2-842c-b350cf68825e/volumes"
Nov 03 13:04:07.652375 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:04:07.652333    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:04:29.745349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.745303    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"]
Nov 03 13:04:29.745980 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.745963    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy"
Nov 03 13:04:29.746055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.745980    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy"
Nov 03 13:04:29.746055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.746001    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy"
Nov 03 13:04:29.746055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.746006    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy"
Nov 03 13:04:29.746218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.746097    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy"
Nov 03 13:04:29.746218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.746107    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed666c5a-f22a-44f2-842c-b350cf68825e" containerName="istio-proxy"
Nov 03 13:04:29.750008 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.749991    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.902162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.804928    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5e9ee9-openshift-default-dockercfg-lpz55\""
Nov 03 13:04:29.902162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.897820    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"]
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.913660    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.913743    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.913893    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.913940    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.913987    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.914070    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.914099    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6dh\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.914150    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:29.933875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:29.914190    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.015557    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.037892    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.037991    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038001    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038079    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038211    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038247    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038281    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038379    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.038824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.038410    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6dh\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.039544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.039118    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.039610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.039587    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.039901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.039872    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.040170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.040146    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.041425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.041406    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.042884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.042863    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.091722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.091697    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6dh\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.091722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.091720    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token\") pod \"e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") " pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.364223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.364133    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:30.804800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.804770    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"]
Nov 03 13:04:30.806017 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:04:30.805988    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213e49b5_465a_480e_955e_c6b83f42194b.slice/crio-524451c71e6acbc4696ae5c537c91264404ee67fa9fe7dec6d3de998b7dcb484 WatchSource:0}: Error finding container 524451c71e6acbc4696ae5c537c91264404ee67fa9fe7dec6d3de998b7dcb484: Status 404 returned error can't find the container with id 524451c71e6acbc4696ae5c537c91264404ee67fa9fe7dec6d3de998b7dcb484
Nov 03 13:04:30.829997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.829554    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:04:30.829997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.829641    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:04:30.829997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:30.829684    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:04:31.300797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:31.300738    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" event={"ID":"213e49b5-465a-480e-955e-c6b83f42194b","Type":"ContainerStarted","Data":"ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"}
Nov 03 13:04:31.301017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:31.300824    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" event={"ID":"213e49b5-465a-480e-955e-c6b83f42194b","Type":"ContainerStarted","Data":"524451c71e6acbc4696ae5c537c91264404ee67fa9fe7dec6d3de998b7dcb484"}
Nov 03 13:04:31.364637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:31.364604    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:31.370662 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:31.370640    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:31.423051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:31.422988    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podStartSLOduration=2.422969622 podStartE2EDuration="2.422969622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:31.419218884 +0000 UTC m=+4412.990802826" watchObservedRunningTime="2026-11-03 13:04:31.422969622 +0000 UTC m=+4412.994553534"
Nov 03 13:04:32.318248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:32.318216    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:32.332344 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:32.332312    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:04:37.570424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.570381    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"]
Nov 03 13:04:37.574707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.574684    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.692771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.653463    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5jn\" (UniqueName: \"kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn\") pod \"payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.732775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.710050    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.733313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.681413    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5e9ee9-dockercfg-wsch7\""
Nov 03 13:04:37.733556 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.681317    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5e9ee9\""
Nov 03 13:04:37.776360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.733502    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"]
Nov 03 13:04:37.776683 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.776662    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"]
Nov 03 13:04:37.786136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.786110    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:37.806412 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.806385    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"]
Nov 03 13:04:37.807804 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:04:37.807777    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:04:37.835023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.834941    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.835162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.835089    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:37.835162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.835152    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhd9x\" (UniqueName: \"kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x\") pod \"payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:37.835357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.835329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5jn\" (UniqueName: \"kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn\") pod \"payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.835814 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.835791    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.889135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.889108    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5jn\" (UniqueName: \"kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn\") pod \"payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.891800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.891775    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:37.937015 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.936982    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:37.937246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.937035    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhd9x\" (UniqueName: \"kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x\") pod \"payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:37.937742 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.937716    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:37.992181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:37.991417    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhd9x\" (UniqueName: \"kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x\") pod \"payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:38.105095 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.105013    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:38.268342 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.268314    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"]
Nov 03 13:04:38.269924 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:04:38.269897    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f821b8_0b47_4e38_95c8_66bf651ca945.slice/crio-bac8a78d5f240bcc65514f5cab58f3d844ce8d1d10c8b6c4ed2374077b81b567 WatchSource:0}: Error finding container bac8a78d5f240bcc65514f5cab58f3d844ce8d1d10c8b6c4ed2374077b81b567: Status 404 returned error can't find the container with id bac8a78d5f240bcc65514f5cab58f3d844ce8d1d10c8b6c4ed2374077b81b567
Nov 03 13:04:38.362239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.362162    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" event={"ID":"56f821b8-0b47-4e38-95c8-66bf651ca945","Type":"ContainerStarted","Data":"c2656edf247f37b4772fd0f6ed800b497f626c366c99ece96de947b43916d103"}
Nov 03 13:04:38.362239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.362218    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" event={"ID":"56f821b8-0b47-4e38-95c8-66bf651ca945","Type":"ContainerStarted","Data":"bac8a78d5f240bcc65514f5cab58f3d844ce8d1d10c8b6c4ed2374077b81b567"}
Nov 03 13:04:38.362428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.362242    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:38.433918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.433882    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:04:38.434176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.433941    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:04:38.469183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.469127    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" podStartSLOduration=1.469113664 podStartE2EDuration="1.469113664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:38.467977341 +0000 UTC m=+4420.039561255" watchObservedRunningTime="2026-11-03 13:04:38.469113664 +0000 UTC m=+4420.040697576"
Nov 03 13:04:38.480403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:38.480374    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"]
Nov 03 13:04:38.483809 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:04:38.483782    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a06299_4dd7_46ef_bcb8_8bd6f00cd217.slice/crio-7fb673ca26c670981b78b772b6e423a95aeaff56e9d48e586843765086d25f99 WatchSource:0}: Error finding container 7fb673ca26c670981b78b772b6e423a95aeaff56e9d48e586843765086d25f99: Status 404 returned error can't find the container with id 7fb673ca26c670981b78b772b6e423a95aeaff56e9d48e586843765086d25f99
Nov 03 13:04:39.262350 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.262319    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"]
Nov 03 13:04:39.266412 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.266394    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.319910 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.319883    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"]
Nov 03 13:04:39.356636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.356599    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.356830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.356645    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.356830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.356732    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wqk\" (UniqueName: \"kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.361859 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.361840    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:04:39.361859 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.361842    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:04:39.361993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.361913    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:04:39.361993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.361952    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:04:39.370130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.370110    4546 generic.go:356] "Generic (PLEG): container finished" podID="fe8e7a74-be1b-4be1-9a28-330448552921" containerID="2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a" exitCode=0
Nov 03 13:04:39.370226 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.370180    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" event={"ID":"fe8e7a74-be1b-4be1-9a28-330448552921","Type":"ContainerDied","Data":"2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"}
Nov 03 13:04:39.370565 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.370547    4546 scope.go:122] "RemoveContainer" containerID="2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"
Nov 03 13:04:39.372027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.372006    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" event={"ID":"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217","Type":"ContainerStarted","Data":"6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6"}
Nov 03 13:04:39.372138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.372044    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" event={"ID":"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217","Type":"ContainerStarted","Data":"7fb673ca26c670981b78b772b6e423a95aeaff56e9d48e586843765086d25f99"}
Nov 03 13:04:39.372268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.372253    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:04:39.372341 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.372269    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:04:39.372341 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.372336    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:04:39.372448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.372370    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:04:39.457480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.457459    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.457568 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.457506    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.457620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.457568    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wqk\" (UniqueName: \"kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.458344 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.458321    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.460349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.460319    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.510860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.510796    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wqk\" (UniqueName: \"kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk\") pod \"maas-api-revoke-sub-bc7bad93-tqhqd\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.580269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.580188    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:39.592846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.592696    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" podStartSLOduration=2.592676683 podStartE2EDuration="2.592676683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:39.589741616 +0000 UTC m=+4421.161325552" watchObservedRunningTime="2026-11-03 13:04:39.592676683 +0000 UTC m=+4421.164260596"
Nov 03 13:04:39.969783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:39.969743    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"]
Nov 03 13:04:39.971964 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:04:39.971940    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c65f30_34fd_4b0c_bc5d_bbdb49c5881c.slice/crio-86d68f78b69c4fe14d5424f0f2cc6d11ea7c6b02c89efd7cfa8a8c34d582fda0 WatchSource:0}: Error finding container 86d68f78b69c4fe14d5424f0f2cc6d11ea7c6b02c89efd7cfa8a8c34d582fda0: Status 404 returned error can't find the container with id 86d68f78b69c4fe14d5424f0f2cc6d11ea7c6b02c89efd7cfa8a8c34d582fda0
Nov 03 13:04:40.380949 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:40.380906    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" event={"ID":"fe8e7a74-be1b-4be1-9a28-330448552921","Type":"ContainerStarted","Data":"38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591"}
Nov 03 13:04:40.382287 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:40.382255    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd" event={"ID":"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c","Type":"ContainerStarted","Data":"86d68f78b69c4fe14d5424f0f2cc6d11ea7c6b02c89efd7cfa8a8c34d582fda0"}
Nov 03 13:04:40.528835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:40.528480    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:04:40.529056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:40.528865    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 13:04:40.529056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:40.528928    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 13:04:41.390055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:41.389953    4546 generic.go:356] "Generic (PLEG): container finished" podID="38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" containerID="9ce263a18f41c5f02848301f22225df9dbb1ecdbe9889e14be9ef5a56f9a95bd" exitCode=0
Nov 03 13:04:41.390542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:41.390048    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd" event={"ID":"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c","Type":"ContainerDied","Data":"9ce263a18f41c5f02848301f22225df9dbb1ecdbe9889e14be9ef5a56f9a95bd"}
Nov 03 13:04:41.527776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:41.524898    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 13:04:41.527776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:41.524963    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 13:04:42.516303 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.515848    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 13:04:42.516303 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.515910    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 13:04:42.604991 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.604957    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:42.703374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.703284    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk\" (UniqueName: \"kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk\") pod \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") "
Nov 03 13:04:42.703543 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.703383    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca\") pod \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") "
Nov 03 13:04:42.703543 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.703440    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca\") pod \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\" (UID: \"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c\") "
Nov 03 13:04:42.703762 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.703721    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca" pod "38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" (UID: "38c65f30-34fd-4b0c-bc5d-bbdb49c5881c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:04:42.703950 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.703933    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:04:42.705640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.705614    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk" pod "38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" (UID: "38c65f30-34fd-4b0c-bc5d-bbdb49c5881c"). InnerVolumeSpecName "kube-api-access-h4wqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:04:42.705864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.705848    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca" pod "38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" (UID: "38c65f30-34fd-4b0c-bc5d-bbdb49c5881c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:04:42.804635 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.804600    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:04:42.804635 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:42.804631    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wqk\" (UniqueName: \"kubernetes.io/projected/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c-kube-api-access-h4wqk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:04:43.404854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:43.404828    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"
Nov 03 13:04:43.404854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:43.404846    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd" event={"ID":"38c65f30-34fd-4b0c-bc5d-bbdb49c5881c","Type":"ContainerDied","Data":"86d68f78b69c4fe14d5424f0f2cc6d11ea7c6b02c89efd7cfa8a8c34d582fda0"}
Nov 03 13:04:43.405132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:43.404894    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d68f78b69c4fe14d5424f0f2cc6d11ea7c6b02c89efd7cfa8a8c34d582fda0"
Nov 03 13:04:43.531962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:43.531917    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 13:04:43.532474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:43.531989    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 13:04:44.518778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:44.512812    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 13:04:44.518778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:44.512875    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 13:04:45.528775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:45.519825    4546 patch_prober.go:29] interesting pod/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 13:04:45.528775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:45.519889    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 13:04:46.513611 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:46.513579    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:04:46.514028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:46.514008    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:04:46.514646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:46.514630    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:04:49.376029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:49.375998    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:04:50.385372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:04:50.385343    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:05:07.750667 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:05:07.750468    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:05:08.456773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:08.454915    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7m2kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:05:08.456773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:08.455007    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:05:37.656122 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:05:37.656082    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:05:43.043379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:43.043347    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"]
Nov 03 13:05:43.089423 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:43.089376    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a850b9f2-pwjh7"]
Nov 03 13:05:44.573605 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:44.573561    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f24dcba-d600-4f4c-b363-cf75a2fa0cc9" path="/var/lib/kubelet/pods/9f24dcba-d600-4f4c-b363-cf75a2fa0cc9/volumes"
Nov 03 13:05:54.042743 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:54.042711    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"]
Nov 03 13:05:54.087936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:54.087905    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4be8bc-6fvk5"]
Nov 03 13:05:54.573852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:05:54.573820    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64c2197-92ca-4ad3-a80b-4383f3c26c5f" path="/var/lib/kubelet/pods/e64c2197-92ca-4ad3-a80b-4383f3c26c5f/volumes"
Nov 03 13:06:07.651576 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:06:07.651534    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:06:08.042807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:08.042774    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"]
Nov 03 13:06:08.088866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:08.088837    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-764464d0-j489n"]
Nov 03 13:06:08.574598 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:08.574568    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083f6dcf-8c1d-419f-a30d-08c0892f1274" path="/var/lib/kubelet/pods/083f6dcf-8c1d-419f-a30d-08c0892f1274/volumes"
Nov 03 13:06:21.589601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.589562    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"]
Nov 03 13:06:21.590491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.590469    4546 cpu_manager.go:414] "containerMap: removing container" podUID="38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" containerName="revoke-keys"
Nov 03 13:06:21.590603 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.590493    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" containerName="revoke-keys"
Nov 03 13:06:21.590669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.590645    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" containerName="revoke-keys"
Nov 03 13:06:21.594341 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.594321    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.648451 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.648419    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"]
Nov 03 13:06:21.689983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.689956    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:06:21.689983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.689954    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:06:21.689983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.689954    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:06:21.690256 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.690048    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:06:21.698660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.698632    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.698829 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.698803    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd8p\" (UniqueName: \"kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.698936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.698921    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.800609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.800575    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.800819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.800689    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd8p\" (UniqueName: \"kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.800819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.800797    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.801483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.801422    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.803275 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.803251    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.853500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.853436    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd8p\" (UniqueName: \"kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p\") pod \"maas-api-revoke-sub-93f9cf62-xh564\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:21.928018 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:21.927979    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:22.304792 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:06:22.304743    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6f7ce7_78fc_40bd_8ea6_ee7229087769.slice/crio-abbb5b83b96652dbd9497dc2a6fe742ad885fe41542fc2b218d00e235478cac4 WatchSource:0}: Error finding container abbb5b83b96652dbd9497dc2a6fe742ad885fe41542fc2b218d00e235478cac4: Status 404 returned error can't find the container with id abbb5b83b96652dbd9497dc2a6fe742ad885fe41542fc2b218d00e235478cac4
Nov 03 13:06:22.306136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:22.306106    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"]
Nov 03 13:06:23.188201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:23.188162    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564" event={"ID":"6b6f7ce7-78fc-40bd-8ea6-ee7229087769","Type":"ContainerStarted","Data":"22a8f958765f6ab8dfdaff393dca9d6556980503df6adad01e1e4529c7c3b029"}
Nov 03 13:06:23.188613 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:23.188224    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564" event={"ID":"6b6f7ce7-78fc-40bd-8ea6-ee7229087769","Type":"ContainerStarted","Data":"abbb5b83b96652dbd9497dc2a6fe742ad885fe41542fc2b218d00e235478cac4"}
Nov 03 13:06:23.296576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:23.296518    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564" podStartSLOduration=2.296500381 podStartE2EDuration="2.296500381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:06:23.292388963 +0000 UTC m=+4524.863973053" watchObservedRunningTime="2026-11-03 13:06:23.296500381 +0000 UTC m=+4524.868084366"
Nov 03 13:06:24.197701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:24.197666    4546 generic.go:356] "Generic (PLEG): container finished" podID="6b6f7ce7-78fc-40bd-8ea6-ee7229087769" containerID="22a8f958765f6ab8dfdaff393dca9d6556980503df6adad01e1e4529c7c3b029" exitCode=0
Nov 03 13:06:24.198157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:24.197769    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564" event={"ID":"6b6f7ce7-78fc-40bd-8ea6-ee7229087769","Type":"ContainerDied","Data":"22a8f958765f6ab8dfdaff393dca9d6556980503df6adad01e1e4529c7c3b029"}
Nov 03 13:06:25.372400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.372372    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:25.475550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.475519    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca\") pod \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") "
Nov 03 13:06:25.475770 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.475587    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p\" (UniqueName: \"kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p\") pod \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") "
Nov 03 13:06:25.475770 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.475635    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca\") pod \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\" (UID: \"6b6f7ce7-78fc-40bd-8ea6-ee7229087769\") "
Nov 03 13:06:25.476195 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.476158    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca" pod "6b6f7ce7-78fc-40bd-8ea6-ee7229087769" (UID: "6b6f7ce7-78fc-40bd-8ea6-ee7229087769"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:06:25.477934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.477905    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca" pod "6b6f7ce7-78fc-40bd-8ea6-ee7229087769" (UID: "6b6f7ce7-78fc-40bd-8ea6-ee7229087769"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:06:25.478075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.478055    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p" pod "6b6f7ce7-78fc-40bd-8ea6-ee7229087769" (UID: "6b6f7ce7-78fc-40bd-8ea6-ee7229087769"). InnerVolumeSpecName "kube-api-access-5kd8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:06:25.577608 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.577572    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:25.577608 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.577607    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kd8p\" (UniqueName: \"kubernetes.io/projected/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-kube-api-access-5kd8p\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:25.577608 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:25.577618    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7ce7-78fc-40bd-8ea6-ee7229087769-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:06:26.213870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.213846    4546 generic.go:356] "Generic (PLEG): container finished" podID="213e49b5-465a-480e-955e-c6b83f42194b" containerID="ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d" exitCode=0
Nov 03 13:06:26.214086 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.213915    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" event={"ID":"213e49b5-465a-480e-955e-c6b83f42194b","Type":"ContainerDied","Data":"ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"}
Nov 03 13:06:26.214438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.214407    4546 scope.go:122] "RemoveContainer" containerID="ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"
Nov 03 13:06:26.215547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.215525    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564" event={"ID":"6b6f7ce7-78fc-40bd-8ea6-ee7229087769","Type":"ContainerDied","Data":"abbb5b83b96652dbd9497dc2a6fe742ad885fe41542fc2b218d00e235478cac4"}
Nov 03 13:06:26.215638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.215573    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abbb5b83b96652dbd9497dc2a6fe742ad885fe41542fc2b218d00e235478cac4"
Nov 03 13:06:26.215638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.215528    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"
Nov 03 13:06:26.216254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.216225    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:06:26.216359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.216294    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:06:26.216359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:26.216330    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:06:27.239620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:27.239579    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" event={"ID":"213e49b5-465a-480e-955e-c6b83f42194b","Type":"ContainerStarted","Data":"dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071"}
Nov 03 13:06:27.364507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:27.364475    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:06:27.364691 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:27.364643    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 13:06:27.364743 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:27.364688    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 13:06:28.365505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:28.365465    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 13:06:28.365940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:28.365529    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 13:06:29.365134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:29.365095    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 13:06:29.365308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:29.365168    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 13:06:30.364667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.364620    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 13:06:30.365228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.364682    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 13:06:30.544400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.544370    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"]
Nov 03 13:06:30.545139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.545116    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6b6f7ce7-78fc-40bd-8ea6-ee7229087769" containerName="revoke-keys"
Nov 03 13:06:30.545139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.545135    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6f7ce7-78fc-40bd-8ea6-ee7229087769" containerName="revoke-keys"
Nov 03 13:06:30.545312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.545276    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6f7ce7-78fc-40bd-8ea6-ee7229087769" containerName="revoke-keys"
Nov 03 13:06:30.548956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.548939    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.571302 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.571251    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.571504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.571351    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.571504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.571386    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmm47\" (UniqueName: \"kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.571504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.571489    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.571711 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.571634    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.571817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.571709    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.638549 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.638466    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"]
Nov 03 13:06:30.645912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.645883    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"openshift-service-ca.crt\""
Nov 03 13:06:30.646090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.645960    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"default-dockercfg-rrntn\""
Nov 03 13:06:30.646090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.645994    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"kube-root-ca.crt\""
Nov 03 13:06:30.646090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.646080    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8570a0\"/\"rate-test-model-8570a0-kserve-self-signed-certs\""
Nov 03 13:06:30.673193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673161    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673199    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmm47\" (UniqueName: \"kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673439 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673377    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673528    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673617    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673705    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673775    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.673881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.673865    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.674094 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.674075    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.675994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.675973    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.676288 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.676269    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.742709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.733970    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmm47\" (UniqueName: \"kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47\") pod \"rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") " pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:30.861989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:30.861954    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:31.258181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:31.258154    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"]
Nov 03 13:06:31.259111 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:06:31.259087    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c9283de_81a2_4464_beef_704698c0148f.slice/crio-e8bf5da966e6bec97df3f0d399e53f3ac2e6bfccd170146b656c53c0225740ab WatchSource:0}: Error finding container e8bf5da966e6bec97df3f0d399e53f3ac2e6bfccd170146b656c53c0225740ab: Status 404 returned error can't find the container with id e8bf5da966e6bec97df3f0d399e53f3ac2e6bfccd170146b656c53c0225740ab
Nov 03 13:06:31.267681 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:31.267653    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d" event={"ID":"6c9283de-81a2-4464-beef-704698c0148f","Type":"ContainerStarted","Data":"e8bf5da966e6bec97df3f0d399e53f3ac2e6bfccd170146b656c53c0225740ab"}
Nov 03 13:06:31.365424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:31.365389    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 13:06:31.365784 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:31.365447    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 13:06:32.278925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:32.278877    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d" event={"ID":"6c9283de-81a2-4464-beef-704698c0148f","Type":"ContainerStarted","Data":"9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d"}
Nov 03 13:06:32.318531 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:32.318495    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:06:32.365631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:32.365600    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 13:06:32.366080 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:32.365670    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 13:06:32.386434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:32.386382    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d" podStartSLOduration=2.386364607 podStartE2EDuration="2.386364607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:06:32.384601331 +0000 UTC m=+4533.956185261" watchObservedRunningTime="2026-11-03 13:06:32.386364607 +0000 UTC m=+4533.957948519"
Nov 03 13:06:32.862839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:32.862810    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:33.371898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:33.371865    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:06:33.373876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:33.373854    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:06:37.826835 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:06:37.826324    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:06:40.862422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:40.862392    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:40.874997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:40.874972    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:41.411569 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:41.411540    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:06:49.044063 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:49.044026    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"]
Nov 03 13:06:49.089847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:49.089820    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62eb92a-lmq8z"]
Nov 03 13:06:50.573917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:06:50.573887    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20878294-56a9-44e3-826b-0bc2aee496ab" path="/var/lib/kubelet/pods/20878294-56a9-44e3-826b-0bc2aee496ab/volumes"
Nov 03 13:07:02.052862 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:07:02.052831    4546 scope.go:122] "RemoveContainer" containerID="4c460b7d74a447228d2154c5d6167cbd21a23c29e1263fe9d7f875fa26f3ba6b"
Nov 03 13:07:02.063001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:07:02.062977    4546 scope.go:122] "RemoveContainer" containerID="bfcef184bcd86f810d16c018d0cd237f1f15fd894c54e6ebe70f3a1b06d1d390"
Nov 03 13:07:07.655496 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:07:07.655460    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:07:14.603888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:07:14.603840    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:07:14.613100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:07:14.609915    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 13:07:14.650880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:07:14.650832    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded"
Nov 03 13:07:37.654615 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:07:37.654579    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:08:02.090574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:02.090541    4546 scope.go:122] "RemoveContainer" containerID="f74ff5dc639c2793c4f50b80a84e07a3b7efc2a16dffad37fe13a7f70ccabe2e"
Nov 03 13:08:02.100227 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:02.100200    4546 scope.go:122] "RemoveContainer" containerID="2601901dc25dc03e179d9e2fe6c433a0858729c20812ff2e6933813abcdac916"
Nov 03 13:08:03.090579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.090538    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"]
Nov 03 13:08:03.096411 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.096387    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.147978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.147951    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"]
Nov 03 13:08:03.169871 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.169837    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.170068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.169996    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8hz\" (UniqueName: \"kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.170157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.170111    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.190195 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.190146    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:08:03.190195 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.190181    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:08:03.211468 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.211441    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:08:03.211618 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.211604    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:08:03.271421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.271378    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.271596 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.271540    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.271673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.271610    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8hz\" (UniqueName: \"kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.272044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.272022    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.274092 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.274070    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.326741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.326705    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8hz\" (UniqueName: \"kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz\") pod \"maas-api-revoke-sub-b402cf6f-rz7js\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.409875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.409797    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:03.809431 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:03.809395    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"]
Nov 03 13:08:04.045901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:04.045830    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js" event={"ID":"45988ae0-773f-408c-b984-24064217cf8f","Type":"ContainerStarted","Data":"ca3d2abe36a52921ee358822ead9275ff582da52c50a4167318702d09db9013c"}
Nov 03 13:08:04.668188 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:04.668154    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"]
Nov 03 13:08:04.668648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:04.668360    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy" containerID="cri-o://38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591" gracePeriod=30
Nov 03 13:08:05.054930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:05.054895    4546 generic.go:356] "Generic (PLEG): container finished" podID="45988ae0-773f-408c-b984-24064217cf8f" containerID="879f4b96a45f9f5b39381c490fcafd9a3ce59034d6dfda06f6f2f3d0af4db292" exitCode=0
Nov 03 13:08:05.055103 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:05.054978    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js" event={"ID":"45988ae0-773f-408c-b984-24064217cf8f","Type":"ContainerDied","Data":"879f4b96a45f9f5b39381c490fcafd9a3ce59034d6dfda06f6f2f3d0af4db292"}
Nov 03 13:08:06.216855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.216829    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:06.332516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.332425    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca\") pod \"45988ae0-773f-408c-b984-24064217cf8f\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") "
Nov 03 13:08:06.332516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.332499    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca\") pod \"45988ae0-773f-408c-b984-24064217cf8f\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") "
Nov 03 13:08:06.332788 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.332539    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz\" (UniqueName: \"kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz\") pod \"45988ae0-773f-408c-b984-24064217cf8f\" (UID: \"45988ae0-773f-408c-b984-24064217cf8f\") "
Nov 03 13:08:06.333128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.333099    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca" pod "45988ae0-773f-408c-b984-24064217cf8f" (UID: "45988ae0-773f-408c-b984-24064217cf8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:08:06.333303 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.333282    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45988ae0-773f-408c-b984-24064217cf8f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:06.335912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.335883    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz" pod "45988ae0-773f-408c-b984-24064217cf8f" (UID: "45988ae0-773f-408c-b984-24064217cf8f"). InnerVolumeSpecName "kube-api-access-4r8hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:06.336003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.335978    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca" pod "45988ae0-773f-408c-b984-24064217cf8f" (UID: "45988ae0-773f-408c-b984-24064217cf8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:06.434414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.434381    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45988ae0-773f-408c-b984-24064217cf8f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:06.434414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:06.434412    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r8hz\" (UniqueName: \"kubernetes.io/projected/45988ae0-773f-408c-b984-24064217cf8f-kube-api-access-4r8hz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:07.069332 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:07.069306    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"
Nov 03 13:08:07.069525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:07.069340    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js" event={"ID":"45988ae0-773f-408c-b984-24064217cf8f","Type":"ContainerDied","Data":"ca3d2abe36a52921ee358822ead9275ff582da52c50a4167318702d09db9013c"}
Nov 03 13:08:07.069525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:07.069391    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3d2abe36a52921ee358822ead9275ff582da52c50a4167318702d09db9013c"
Nov 03 13:08:07.652288 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:08:07.652203    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:08:09.953201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:09.953142    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:08:10.083013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.082982    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083091    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083120    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083146    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083218    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083251    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083347    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083380    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083418    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token\") pod \"fe8e7a74-be1b-4be1-9a28-330448552921\" (UID: \"fe8e7a74-be1b-4be1-9a28-330448552921\") "
Nov 03 13:08:10.083452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083430    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:10.083695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083452    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:10.083695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083467    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:10.083695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083561    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:08:10.083886 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.083864    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:10.084043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.084020    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.084385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.084367    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.084507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.084483    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.084628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.084509    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.084628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.084526    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8e7a74-be1b-4be1-9a28-330448552921-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.085956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.085921    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:10.086058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.086004    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:08:10.086193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.086173    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "kube-api-access-dmgv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:10.086245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.086176    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy" pod "fe8e7a74-be1b-4be1-9a28-330448552921" (UID: "fe8e7a74-be1b-4be1-9a28-330448552921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:10.093655 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.093633    4546 generic.go:356] "Generic (PLEG): container finished" podID="fe8e7a74-be1b-4be1-9a28-330448552921" containerID="38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591" exitCode=0
Nov 03 13:08:10.093778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.093734    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"
Nov 03 13:08:10.093828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.093772    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" event={"ID":"fe8e7a74-be1b-4be1-9a28-330448552921","Type":"ContainerDied","Data":"38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591"}
Nov 03 13:08:10.093865 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.093829    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl" event={"ID":"fe8e7a74-be1b-4be1-9a28-330448552921","Type":"ContainerDied","Data":"6d69dd6229f8b7971ebef5164f2899ea09ce0abd1f9723d39872db5975b50573"}
Nov 03 13:08:10.093865 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.093852    4546 scope.go:122] "RemoveContainer" containerID="38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591"
Nov 03 13:08:10.117475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.117456    4546 scope.go:122] "RemoveContainer" containerID="2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"
Nov 03 13:08:10.127164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.127139    4546 scope.go:122] "RemoveContainer" containerID="38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591"
Nov 03 13:08:10.127462 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:08:10.127441    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591\": container with ID starting with 38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591 not found: ID does not exist" containerID="38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591"
Nov 03 13:08:10.127565 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.127468    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591"} err="failed to get container status \"38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591\": rpc error: code = NotFound desc = could not find container \"38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591\": container with ID starting with 38fb860f6eefe7a8f4e7e33579c4aeb24c30be25c57991125aa2c4c3d704d591 not found: ID does not exist"
Nov 03 13:08:10.127565 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.127497    4546 scope.go:122] "RemoveContainer" containerID="2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"
Nov 03 13:08:10.127833 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:08:10.127802    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a\": container with ID starting with 2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a not found: ID does not exist" containerID="2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"
Nov 03 13:08:10.127931 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.127843    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a"} err="failed to get container status \"2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a\": rpc error: code = NotFound desc = could not find container \"2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a\": container with ID starting with 2da1f0f6748deedf2a06d12b6917226e6c043fad78df1b927b9aebc64e34be3a not found: ID does not exist"
Nov 03 13:08:10.186166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.186141    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8e7a74-be1b-4be1-9a28-330448552921-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.186295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.186175    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.186295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.186186    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8e7a74-be1b-4be1-9a28-330448552921-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.186295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.186198    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmgv5\" (UniqueName: \"kubernetes.io/projected/fe8e7a74-be1b-4be1-9a28-330448552921-kube-api-access-dmgv5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:10.261907 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.261879    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"]
Nov 03 13:08:10.265086 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.265065    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6bd346ef-openshift-default-fb84d9949-vlsvl"]
Nov 03 13:08:10.573769 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:10.573670    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" path="/var/lib/kubelet/pods/fe8e7a74-be1b-4be1-9a28-330448552921/volumes"
Nov 03 13:08:11.952381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.952350    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"]
Nov 03 13:08:11.953173 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953152    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy"
Nov 03 13:08:11.953297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953173    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy"
Nov 03 13:08:11.953297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953221    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy"
Nov 03 13:08:11.953297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953230    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy"
Nov 03 13:08:11.953297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953279    4546 cpu_manager.go:414] "containerMap: removing container" podUID="45988ae0-773f-408c-b984-24064217cf8f" containerName="revoke-keys"
Nov 03 13:08:11.953297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953286    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45988ae0-773f-408c-b984-24064217cf8f" containerName="revoke-keys"
Nov 03 13:08:11.953563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953451    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="45988ae0-773f-408c-b984-24064217cf8f" containerName="revoke-keys"
Nov 03 13:08:11.953563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953468    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy"
Nov 03 13:08:11.953563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.953482    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8e7a74-be1b-4be1-9a28-330448552921" containerName="istio-proxy"
Nov 03 13:08:11.958523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:11.958504    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.007300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.007273    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:08:12.007300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.007301    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:08:12.007552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.007464    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:08:12.007552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.007478    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:08:12.011589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.011564    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"]
Nov 03 13:08:12.108492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.108460    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlrk\" (UniqueName: \"kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.108676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.108505    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.108676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.108614    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.210212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.210122    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlrk\" (UniqueName: \"kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.210212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.210170    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.210212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.210212    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.210910 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.210888    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.212702 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.212682    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.263055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.263027    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlrk\" (UniqueName: \"kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk\") pod \"maas-api-revoke-sub-8d16f979-qvjkn\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.271934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.271912    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:12.639603 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:12.639572    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"]
Nov 03 13:08:13.114591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:13.114559    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn" event={"ID":"1175b324-73ae-49be-9a14-955267d71f6b","Type":"ContainerStarted","Data":"1df03c766795b32b70b0f0e7dccce93dd9e984d25f76130eecd93b391ce16585"}
Nov 03 13:08:13.115010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:13.114614    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn" event={"ID":"1175b324-73ae-49be-9a14-955267d71f6b","Type":"ContainerStarted","Data":"f46d9c3cb944a797b921d63d85d6bf9a86aefacb971ae4638b4b1328bba30bd8"}
Nov 03 13:08:13.220867 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:13.220818    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn" podStartSLOduration=2.22080445 podStartE2EDuration="2.22080445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:13.219241833 +0000 UTC m=+4634.790825766" watchObservedRunningTime="2026-11-03 13:08:13.22080445 +0000 UTC m=+4634.792388362"
Nov 03 13:08:14.122068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:14.122036    4546 generic.go:356] "Generic (PLEG): container finished" podID="1175b324-73ae-49be-9a14-955267d71f6b" containerID="1df03c766795b32b70b0f0e7dccce93dd9e984d25f76130eecd93b391ce16585" exitCode=0
Nov 03 13:08:14.122489 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:14.122092    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn" event={"ID":"1175b324-73ae-49be-9a14-955267d71f6b","Type":"ContainerDied","Data":"1df03c766795b32b70b0f0e7dccce93dd9e984d25f76130eecd93b391ce16585"}
Nov 03 13:08:15.297077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.297052    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:15.350860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.350835    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk\" (UniqueName: \"kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk\") pod \"1175b324-73ae-49be-9a14-955267d71f6b\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") "
Nov 03 13:08:15.350999 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.350916    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca\") pod \"1175b324-73ae-49be-9a14-955267d71f6b\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") "
Nov 03 13:08:15.351046 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.351007    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca\") pod \"1175b324-73ae-49be-9a14-955267d71f6b\" (UID: \"1175b324-73ae-49be-9a14-955267d71f6b\") "
Nov 03 13:08:15.351360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.351340    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca" pod "1175b324-73ae-49be-9a14-955267d71f6b" (UID: "1175b324-73ae-49be-9a14-955267d71f6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:08:15.351800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.351778    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:15.353173 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.353151    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca" pod "1175b324-73ae-49be-9a14-955267d71f6b" (UID: "1175b324-73ae-49be-9a14-955267d71f6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:15.353264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.353256    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk" pod "1175b324-73ae-49be-9a14-955267d71f6b" (UID: "1175b324-73ae-49be-9a14-955267d71f6b"). InnerVolumeSpecName "kube-api-access-4mlrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:15.453558 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.453476    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlrk\" (UniqueName: \"kubernetes.io/projected/1175b324-73ae-49be-9a14-955267d71f6b-kube-api-access-4mlrk\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:15.453558 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:15.453513    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1175b324-73ae-49be-9a14-955267d71f6b-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:16.136792 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:16.136744    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn" event={"ID":"1175b324-73ae-49be-9a14-955267d71f6b","Type":"ContainerDied","Data":"f46d9c3cb944a797b921d63d85d6bf9a86aefacb971ae4638b4b1328bba30bd8"}
Nov 03 13:08:16.136792 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:16.136786    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"
Nov 03 13:08:16.137073 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:16.136815    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46d9c3cb944a797b921d63d85d6bf9a86aefacb971ae4638b4b1328bba30bd8"
Nov 03 13:08:16.383397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:16.383364    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"]
Nov 03 13:08:28.523043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.523010    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"]
Nov 03 13:08:28.523630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.523615    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1175b324-73ae-49be-9a14-955267d71f6b" containerName="revoke-keys"
Nov 03 13:08:28.523677 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.523632    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1175b324-73ae-49be-9a14-955267d71f6b" containerName="revoke-keys"
Nov 03 13:08:28.523766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.523741    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1175b324-73ae-49be-9a14-955267d71f6b" containerName="revoke-keys"
Nov 03 13:08:28.527833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.527812    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.576111 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.576080    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:08:28.610962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.610931    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qkb\" (UniqueName: \"kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.611213 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.611192    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.611313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.611220    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.622435 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.622410    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:08:28.622576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.622513    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:08:28.622643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.622589    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:08:28.627226 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.627206    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"]
Nov 03 13:08:28.712026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.711979    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.712026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.712019    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.712314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.712095    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qkb\" (UniqueName: \"kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.712713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.712672    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.714612 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.714585    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.765309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.765274    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qkb\" (UniqueName: \"kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb\") pod \"maas-api-revoke-sub-1ff63217-vv6dh\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:28.843267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:28.843200    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:29.271501 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:29.271447    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"]
Nov 03 13:08:30.453510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:30.453450    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh" event={"ID":"b60d0580-3ec0-48cb-9d8a-aa023c9fea84","Type":"ContainerStarted","Data":"a9c2419ad4d9102ba7b246b7de4e3b9dcb51206e7363b22fffa2d109e3df32a3"}
Nov 03 13:08:30.454202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:30.454155    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh" event={"ID":"b60d0580-3ec0-48cb-9d8a-aa023c9fea84","Type":"ContainerStarted","Data":"0c9284505e04cdbf21113606d06fa6d27b842529fbbbddbec420549c02303c10"}
Nov 03 13:08:30.524960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:30.524838    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt" podUID="9e31e600-1afa-400e-8acb-5c32a3c2179d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:08:30.630777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:30.630147    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh" podStartSLOduration=2.630126809 podStartE2EDuration="2.630126809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:30.596232612 +0000 UTC m=+4652.167816526" watchObservedRunningTime="2026-11-03 13:08:30.630126809 +0000 UTC m=+4652.201710722"
Nov 03 13:08:30.647789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:30.644829    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded"
Nov 03 13:08:30.818801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:30.813855    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:08:31.406239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.406194    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" containerID="cri-o://6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b" gracePeriod=45
Nov 03 13:08:31.462660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.462626    4546 generic.go:356] "Generic (PLEG): container finished" podID="b60d0580-3ec0-48cb-9d8a-aa023c9fea84" containerID="a9c2419ad4d9102ba7b246b7de4e3b9dcb51206e7363b22fffa2d109e3df32a3" exitCode=0
Nov 03 13:08:31.463209 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.462668    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh" event={"ID":"b60d0580-3ec0-48cb-9d8a-aa023c9fea84","Type":"ContainerDied","Data":"a9c2419ad4d9102ba7b246b7de4e3b9dcb51206e7363b22fffa2d109e3df32a3"}
Nov 03 13:08:31.585600 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.585564    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused"
Nov 03 13:08:31.734149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.734124    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 13:08:31.794548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794519    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home\") pod \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") "
Nov 03 13:08:31.794795 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794604    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache\") pod \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") "
Nov 03 13:08:31.794795 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794762    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs\") pod \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") "
Nov 03 13:08:31.794941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794877    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn\" (UniqueName: \"kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn\") pod \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") "
Nov 03 13:08:31.795003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794944    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm\") pod \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") "
Nov 03 13:08:31.795003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794950    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache" pod "d8c714d2-b2d5-4f50-a3f6-5972844d1678" (UID: "d8c714d2-b2d5-4f50-a3f6-5972844d1678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.795003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794870    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home" pod "d8c714d2-b2d5-4f50-a3f6-5972844d1678" (UID: "d8c714d2-b2d5-4f50-a3f6-5972844d1678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.795003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.794985    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir\") pod \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\" (UID: \"d8c714d2-b2d5-4f50-a3f6-5972844d1678\") "
Nov 03 13:08:31.795588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.795562    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.795699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.795592    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.795831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.795806    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir" pod "d8c714d2-b2d5-4f50-a3f6-5972844d1678" (UID: "d8c714d2-b2d5-4f50-a3f6-5972844d1678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.797724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.797697    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm" pod "d8c714d2-b2d5-4f50-a3f6-5972844d1678" (UID: "d8c714d2-b2d5-4f50-a3f6-5972844d1678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.798065 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.797994    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn" pod "d8c714d2-b2d5-4f50-a3f6-5972844d1678" (UID: "d8c714d2-b2d5-4f50-a3f6-5972844d1678"). InnerVolumeSpecName "kube-api-access-v9brn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:31.798881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.798852    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs" pod "d8c714d2-b2d5-4f50-a3f6-5972844d1678" (UID: "d8c714d2-b2d5-4f50-a3f6-5972844d1678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:31.897103 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.897063    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.897103 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.897103    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9brn\" (UniqueName: \"kubernetes.io/projected/d8c714d2-b2d5-4f50-a3f6-5972844d1678-kube-api-access-v9brn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.897381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.897118    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.897381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:31.897131    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c714d2-b2d5-4f50-a3f6-5972844d1678-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:32.471291 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.471255    4546 generic.go:356] "Generic (PLEG): container finished" podID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerID="6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b" exitCode=0
Nov 03 13:08:32.471826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.471316    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"
Nov 03 13:08:32.471826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.471337    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" event={"ID":"d8c714d2-b2d5-4f50-a3f6-5972844d1678","Type":"ContainerDied","Data":"6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b"}
Nov 03 13:08:32.471826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.471382    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx" event={"ID":"d8c714d2-b2d5-4f50-a3f6-5972844d1678","Type":"ContainerDied","Data":"eeb1d6d7b54784b434dec1d834e2c8c3dc883fec448840da48fbc15d98e6bf24"}
Nov 03 13:08:32.471826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.471399    4546 scope.go:122] "RemoveContainer" containerID="6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b"
Nov 03 13:08:32.482497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.482477    4546 scope.go:122] "RemoveContainer" containerID="6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b"
Nov 03 13:08:32.482893 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:08:32.482868    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b\": container with ID starting with 6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b not found: ID does not exist" containerID="6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b"
Nov 03 13:08:32.482988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.482900    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b"} err="failed to get container status \"6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b\": rpc error: code = NotFound desc = could not find container \"6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b\": container with ID starting with 6be34275babc7487f0f3ef27e191eb8d57c522daeb250d7f3c377d4189d6359b not found: ID does not exist"
Nov 03 13:08:32.626857 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.626825    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"]
Nov 03 13:08:32.632282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.632257    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:32.672300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.672266    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/simulator-w4-8d2c7c-kserve-784b9877b4-hhlvx"]
Nov 03 13:08:32.714648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.714618    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca\") pod \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") "
Nov 03 13:08:32.714835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.714709    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca\") pod \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") "
Nov 03 13:08:32.714835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.714797    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb\" (UniqueName: \"kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb\") pod \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\" (UID: \"b60d0580-3ec0-48cb-9d8a-aa023c9fea84\") "
Nov 03 13:08:32.715010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.714987    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca" pod "b60d0580-3ec0-48cb-9d8a-aa023c9fea84" (UID: "b60d0580-3ec0-48cb-9d8a-aa023c9fea84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:08:32.715510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.715485    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:32.717216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.717184    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca" pod "b60d0580-3ec0-48cb-9d8a-aa023c9fea84" (UID: "b60d0580-3ec0-48cb-9d8a-aa023c9fea84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:32.717314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.717213    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb" pod "b60d0580-3ec0-48cb-9d8a-aa023c9fea84" (UID: "b60d0580-3ec0-48cb-9d8a-aa023c9fea84"). InnerVolumeSpecName "kube-api-access-x9qkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:32.816466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.816429    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9qkb\" (UniqueName: \"kubernetes.io/projected/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-kube-api-access-x9qkb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:32.816466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:32.816458    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60d0580-3ec0-48cb-9d8a-aa023c9fea84-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:33.480044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:33.480010    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"
Nov 03 13:08:33.480426 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:33.480010    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh" event={"ID":"b60d0580-3ec0-48cb-9d8a-aa023c9fea84","Type":"ContainerDied","Data":"0c9284505e04cdbf21113606d06fa6d27b842529fbbbddbec420549c02303c10"}
Nov 03 13:08:33.480426 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:33.480141    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9284505e04cdbf21113606d06fa6d27b842529fbbbddbec420549c02303c10"
Nov 03 13:08:34.574017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:08:34.573985    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" path="/var/lib/kubelet/pods/d8c714d2-b2d5-4f50-a3f6-5972844d1678/volumes"
Nov 03 13:08:37.652180 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:08:37.652144    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:09:07.652093 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:09:07.652058    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:09:37.653191 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:09:37.653109    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:09:43.044611 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:09:43.044546    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"]
Nov 03 13:09:43.090300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:09:43.090270    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7bad93-tqhqd"]
Nov 03 13:09:44.573828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:09:44.573796    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c65f30-34fd-4b0c-bc5d-bbdb49c5881c" path="/var/lib/kubelet/pods/38c65f30-34fd-4b0c-bc5d-bbdb49c5881c/volumes"
Nov 03 13:10:07.652083 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:10:07.652045    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:10:37.653476 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:10:37.653439    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:10:40.253157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.253122    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"]
Nov 03 13:10:40.253846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.253829    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b60d0580-3ec0-48cb-9d8a-aa023c9fea84" containerName="revoke-keys"
Nov 03 13:10:40.253906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.253847    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60d0580-3ec0-48cb-9d8a-aa023c9fea84" containerName="revoke-keys"
Nov 03 13:10:40.253906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.253864    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main"
Nov 03 13:10:40.253906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.253871    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main"
Nov 03 13:10:40.254009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.253984    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c714d2-b2d5-4f50-a3f6-5972844d1678" containerName="main"
Nov 03 13:10:40.254009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.254004    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60d0580-3ec0-48cb-9d8a-aa023c9fea84" containerName="revoke-keys"
Nov 03 13:10:40.257442 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.257426    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.311958 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.311925    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"]
Nov 03 13:10:40.325634 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.325604    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvd4\" (UniqueName: \"kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.325830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.325673    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.325830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.325821    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.353778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.353740    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:10:40.353778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.353741    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:10:40.353943 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.353741    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:10:40.353943 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.353742    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:10:40.427312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.427287    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.427447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.427401    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvd4\" (UniqueName: \"kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.427447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.427432    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.428121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.428094    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.429885 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.429862    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.480146 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.480123    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvd4\" (UniqueName: \"kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4\") pod \"maas-api-revoke-sub-80092f4b-n9j8m\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.571207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.571130    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:40.937277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:40.937251    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"]
Nov 03 13:10:40.937990 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:10:40.937965    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa7f2af_4899_4d3b_8159_81a7b19621e3.slice/crio-4d607e550ba2f29ebf6ef5af921ab46d575af54f618f1c0eedfea454ecb01b63 WatchSource:0}: Error finding container 4d607e550ba2f29ebf6ef5af921ab46d575af54f618f1c0eedfea454ecb01b63: Status 404 returned error can't find the container with id 4d607e550ba2f29ebf6ef5af921ab46d575af54f618f1c0eedfea454ecb01b63
Nov 03 13:10:41.337050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:41.337017    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m" event={"ID":"aaa7f2af-4899-4d3b-8159-81a7b19621e3","Type":"ContainerStarted","Data":"381e146514da1d008e978619e25f3e8402a15046077ae2c769852c02376f1d37"}
Nov 03 13:10:41.337451 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:41.337077    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m" event={"ID":"aaa7f2af-4899-4d3b-8159-81a7b19621e3","Type":"ContainerStarted","Data":"4d607e550ba2f29ebf6ef5af921ab46d575af54f618f1c0eedfea454ecb01b63"}
Nov 03 13:10:41.442798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:41.442736    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m" podStartSLOduration=1.44272247 podStartE2EDuration="1.44272247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:41.441796782 +0000 UTC m=+4783.013380713" watchObservedRunningTime="2026-11-03 13:10:41.44272247 +0000 UTC m=+4783.014306382"
Nov 03 13:10:42.345403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:42.345368    4546 generic.go:356] "Generic (PLEG): container finished" podID="aaa7f2af-4899-4d3b-8159-81a7b19621e3" containerID="381e146514da1d008e978619e25f3e8402a15046077ae2c769852c02376f1d37" exitCode=0
Nov 03 13:10:42.345822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:42.345453    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m" event={"ID":"aaa7f2af-4899-4d3b-8159-81a7b19621e3","Type":"ContainerDied","Data":"381e146514da1d008e978619e25f3e8402a15046077ae2c769852c02376f1d37"}
Nov 03 13:10:43.518568 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.518545    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:43.665709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.665619    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4\" (UniqueName: \"kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4\") pod \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") "
Nov 03 13:10:43.665911 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.665714    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca\") pod \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") "
Nov 03 13:10:43.665911 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.665763    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca\") pod \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\" (UID: \"aaa7f2af-4899-4d3b-8159-81a7b19621e3\") "
Nov 03 13:10:43.666122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.666095    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca" pod "aaa7f2af-4899-4d3b-8159-81a7b19621e3" (UID: "aaa7f2af-4899-4d3b-8159-81a7b19621e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:43.666827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.666545    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:43.668177 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.668150    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4" pod "aaa7f2af-4899-4d3b-8159-81a7b19621e3" (UID: "aaa7f2af-4899-4d3b-8159-81a7b19621e3"). InnerVolumeSpecName "kube-api-access-dpvd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:10:43.668416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.668391    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca" pod "aaa7f2af-4899-4d3b-8159-81a7b19621e3" (UID: "aaa7f2af-4899-4d3b-8159-81a7b19621e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:10:43.767777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.767723    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvd4\" (UniqueName: \"kubernetes.io/projected/aaa7f2af-4899-4d3b-8159-81a7b19621e3-kube-api-access-dpvd4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:43.767777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:43.767775    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa7f2af-4899-4d3b-8159-81a7b19621e3-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:44.368968 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:44.368937    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"
Nov 03 13:10:44.369159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:44.368931    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m" event={"ID":"aaa7f2af-4899-4d3b-8159-81a7b19621e3","Type":"ContainerDied","Data":"4d607e550ba2f29ebf6ef5af921ab46d575af54f618f1c0eedfea454ecb01b63"}
Nov 03 13:10:44.369159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:44.369098    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d607e550ba2f29ebf6ef5af921ab46d575af54f618f1c0eedfea454ecb01b63"
Nov 03 13:10:50.848152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.848118    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"]
Nov 03 13:10:50.848776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.848732    4546 cpu_manager.go:414] "containerMap: removing container" podUID="aaa7f2af-4899-4d3b-8159-81a7b19621e3" containerName="revoke-keys"
Nov 03 13:10:50.848776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.848762    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa7f2af-4899-4d3b-8159-81a7b19621e3" containerName="revoke-keys"
Nov 03 13:10:50.848907 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.848898    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa7f2af-4899-4d3b-8159-81a7b19621e3" containerName="revoke-keys"
Nov 03 13:10:50.852574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.852555    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:50.907062 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.907033    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"]
Nov 03 13:10:50.947861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.947833    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:10:50.947861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.947835    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:10:50.948066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.947892    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:10:50.948066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.947915    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:10:50.954694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.954670    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:50.954915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.954899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:50.954962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:50.954932    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7kd\" (UniqueName: \"kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.056177 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.056144    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.056359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.056185    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7kd\" (UniqueName: \"kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.056359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.056250    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.056965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.056942    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.058763 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.058724    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.109339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.109280    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7kd\" (UniqueName: \"kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd\") pod \"maas-api-revoke-sub-de2dfddc-vj5nw\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.165805 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.165774    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:51.536803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:51.536743    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"]
Nov 03 13:10:51.538527 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:10:51.538498    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ddc376_a4a7_4f09_acde_de7ed80c5751.slice/crio-629269006a13962c9dac17a19c7897b084b3ea69fcce6914391c136dab7d3cdf WatchSource:0}: Error finding container 629269006a13962c9dac17a19c7897b084b3ea69fcce6914391c136dab7d3cdf: Status 404 returned error can't find the container with id 629269006a13962c9dac17a19c7897b084b3ea69fcce6914391c136dab7d3cdf
Nov 03 13:10:52.421237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:52.421048    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw" event={"ID":"25ddc376-a4a7-4f09-acde-de7ed80c5751","Type":"ContainerStarted","Data":"1df9011df9f5863aaea627fa9490d3f08a148a846642d27feaa4394fb837364c"}
Nov 03 13:10:52.421237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:52.421111    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw" event={"ID":"25ddc376-a4a7-4f09-acde-de7ed80c5751","Type":"ContainerStarted","Data":"629269006a13962c9dac17a19c7897b084b3ea69fcce6914391c136dab7d3cdf"}
Nov 03 13:10:53.429704 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.429671    4546 generic.go:356] "Generic (PLEG): container finished" podID="25ddc376-a4a7-4f09-acde-de7ed80c5751" containerID="1df9011df9f5863aaea627fa9490d3f08a148a846642d27feaa4394fb837364c" exitCode=0
Nov 03 13:10:53.430616 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.429710    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw" event={"ID":"25ddc376-a4a7-4f09-acde-de7ed80c5751","Type":"ContainerDied","Data":"1df9011df9f5863aaea627fa9490d3f08a148a846642d27feaa4394fb837364c"}
Nov 03 13:10:53.588228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.588205    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:53.694233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.694145    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd\" (UniqueName: \"kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd\") pod \"25ddc376-a4a7-4f09-acde-de7ed80c5751\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") "
Nov 03 13:10:53.694418 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.694258    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca\") pod \"25ddc376-a4a7-4f09-acde-de7ed80c5751\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") "
Nov 03 13:10:53.694485 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.694443    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca\") pod \"25ddc376-a4a7-4f09-acde-de7ed80c5751\" (UID: \"25ddc376-a4a7-4f09-acde-de7ed80c5751\") "
Nov 03 13:10:53.694912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.694884    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca" pod "25ddc376-a4a7-4f09-acde-de7ed80c5751" (UID: "25ddc376-a4a7-4f09-acde-de7ed80c5751"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:10:53.696969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.696942    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca" pod "25ddc376-a4a7-4f09-acde-de7ed80c5751" (UID: "25ddc376-a4a7-4f09-acde-de7ed80c5751"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:10:53.697816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.697787    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd" pod "25ddc376-a4a7-4f09-acde-de7ed80c5751" (UID: "25ddc376-a4a7-4f09-acde-de7ed80c5751"). InnerVolumeSpecName "kube-api-access-jn7kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:10:53.796264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.796224    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:53.796264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.796260    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ddc376-a4a7-4f09-acde-de7ed80c5751-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:53.796545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:53.796275    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7kd\" (UniqueName: \"kubernetes.io/projected/25ddc376-a4a7-4f09-acde-de7ed80c5751-kube-api-access-jn7kd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:10:54.437359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:54.437330    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"
Nov 03 13:10:54.437875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:54.437399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw" event={"ID":"25ddc376-a4a7-4f09-acde-de7ed80c5751","Type":"ContainerDied","Data":"629269006a13962c9dac17a19c7897b084b3ea69fcce6914391c136dab7d3cdf"}
Nov 03 13:10:54.437875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:10:54.437467    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="629269006a13962c9dac17a19c7897b084b3ea69fcce6914391c136dab7d3cdf"
Nov 03 13:11:02.163350 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:02.163316    4546 scope.go:122] "RemoveContainer" containerID="9ce263a18f41c5f02848301f22225df9dbb1ecdbe9889e14be9ef5a56f9a95bd"
Nov 03 13:11:07.652679 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:11:07.652602    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:11:26.044128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:26.044090    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"]
Nov 03 13:11:26.089381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:26.089354    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f9cf62-xh564"]
Nov 03 13:11:26.573655 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:26.573625    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6f7ce7-78fc-40bd-8ea6-ee7229087769" path="/var/lib/kubelet/pods/6b6f7ce7-78fc-40bd-8ea6-ee7229087769/volumes"
Nov 03 13:11:36.983848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:36.983817    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"]
Nov 03 13:11:36.984411 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:36.984398    4546 cpu_manager.go:414] "containerMap: removing container" podUID="25ddc376-a4a7-4f09-acde-de7ed80c5751" containerName="revoke-keys"
Nov 03 13:11:36.984459 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:36.984412    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ddc376-a4a7-4f09-acde-de7ed80c5751" containerName="revoke-keys"
Nov 03 13:11:36.984539 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:36.984528    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ddc376-a4a7-4f09-acde-de7ed80c5751" containerName="revoke-keys"
Nov 03 13:11:36.988308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:36.988288    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.043658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.043624    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"]
Nov 03 13:11:37.079612 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.079581    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.079794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.079648    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztxt\" (UniqueName: \"kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.079864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.079793    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.085427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.085406    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:11:37.085580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.085400    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:11:37.085649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.085575    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:11:37.085649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.085479    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:11:37.180581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.180558    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.180707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.180600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztxt\" (UniqueName: \"kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.180707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.180647    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.181329 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.181304    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.183108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.183088    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.233227 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.233199    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztxt\" (UniqueName: \"kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt\") pod \"maas-api-revoke-sub-86ad9cac-8fft4\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.302355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.302292    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:37.651940 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:11:37.651855    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:11:37.668637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.668609    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"]
Nov 03 13:11:37.670120 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:11:37.670092    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf98a2e_e005_458c_9e6a_87718fe4715b.slice/crio-4206d7557b1d3099af11b7f61e60d2ac5e524379ec03fb384c6f4ad7848d1b56 WatchSource:0}: Error finding container 4206d7557b1d3099af11b7f61e60d2ac5e524379ec03fb384c6f4ad7848d1b56: Status 404 returned error can't find the container with id 4206d7557b1d3099af11b7f61e60d2ac5e524379ec03fb384c6f4ad7848d1b56
Nov 03 13:11:37.845548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:37.845513    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4" event={"ID":"5bf98a2e-e005-458c-9e6a-87718fe4715b","Type":"ContainerStarted","Data":"4206d7557b1d3099af11b7f61e60d2ac5e524379ec03fb384c6f4ad7848d1b56"}
Nov 03 13:11:38.853150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:38.853066    4546 generic.go:356] "Generic (PLEG): container finished" podID="5bf98a2e-e005-458c-9e6a-87718fe4715b" containerID="6c224c26cf85e9a4b6d2005b632f26dd847fc08b61c6007e8272f9ccefee6e4e" exitCode=0
Nov 03 13:11:38.853150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:38.853139    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4" event={"ID":"5bf98a2e-e005-458c-9e6a-87718fe4715b","Type":"ContainerDied","Data":"6c224c26cf85e9a4b6d2005b632f26dd847fc08b61c6007e8272f9ccefee6e4e"}
Nov 03 13:11:40.019937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.019911    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:40.107300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.107275    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca\") pod \"5bf98a2e-e005-458c-9e6a-87718fe4715b\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") "
Nov 03 13:11:40.107497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.107393    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt\" (UniqueName: \"kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt\") pod \"5bf98a2e-e005-458c-9e6a-87718fe4715b\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") "
Nov 03 13:11:40.107497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.107426    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca\") pod \"5bf98a2e-e005-458c-9e6a-87718fe4715b\" (UID: \"5bf98a2e-e005-458c-9e6a-87718fe4715b\") "
Nov 03 13:11:40.107671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.107648    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca" pod "5bf98a2e-e005-458c-9e6a-87718fe4715b" (UID: "5bf98a2e-e005-458c-9e6a-87718fe4715b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:11:40.107911 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.107893    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:40.109556 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.109535    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt" pod "5bf98a2e-e005-458c-9e6a-87718fe4715b" (UID: "5bf98a2e-e005-458c-9e6a-87718fe4715b"). InnerVolumeSpecName "kube-api-access-dztxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:11:40.109679 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.109661    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca" pod "5bf98a2e-e005-458c-9e6a-87718fe4715b" (UID: "5bf98a2e-e005-458c-9e6a-87718fe4715b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:11:40.208798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.208739    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztxt\" (UniqueName: \"kubernetes.io/projected/5bf98a2e-e005-458c-9e6a-87718fe4715b-kube-api-access-dztxt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:40.208798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.208793    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bf98a2e-e005-458c-9e6a-87718fe4715b-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:11:40.867721 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.867626    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"
Nov 03 13:11:40.867721 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.867629    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4" event={"ID":"5bf98a2e-e005-458c-9e6a-87718fe4715b","Type":"ContainerDied","Data":"4206d7557b1d3099af11b7f61e60d2ac5e524379ec03fb384c6f4ad7848d1b56"}
Nov 03 13:11:40.867721 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:11:40.867690    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4206d7557b1d3099af11b7f61e60d2ac5e524379ec03fb384c6f4ad7848d1b56"
Nov 03 13:12:07.652321 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:12:07.652281    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:12:09.382045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.382014    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"]
Nov 03 13:12:09.382625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.382610    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5bf98a2e-e005-458c-9e6a-87718fe4715b" containerName="revoke-keys"
Nov 03 13:12:09.382625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.382626    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf98a2e-e005-458c-9e6a-87718fe4715b" containerName="revoke-keys"
Nov 03 13:12:09.382805 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.382787    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf98a2e-e005-458c-9e6a-87718fe4715b" containerName="revoke-keys"
Nov 03 13:12:09.386363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.386345    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.479174 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.479144    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.479348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.479192    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.479404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.479381    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2mf\" (UniqueName: \"kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.482510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.482465    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:12:09.482510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.482495    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:12:09.482510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.482509    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:12:09.482724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.482564    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:12:09.486224 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.486201    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"]
Nov 03 13:12:09.580444 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.580413    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.580590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.580458    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.580661 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.580596    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2mf\" (UniqueName: \"kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.581114 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.581093    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.583199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.583178    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.634151 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.634095    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2mf\" (UniqueName: \"kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf\") pod \"maas-api-revoke-sub-b024024e-9n4hz\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:09.699916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:09.699888    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:10.070908 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:10.070878    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"]
Nov 03 13:12:10.071386 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:12:10.071353    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373ee556_9550_482f_8806_71b0d7147ba1.slice/crio-0595898a50927506a23fe30fa03033d5c264819613ff20de18bcd962e3e8e234 WatchSource:0}: Error finding container 0595898a50927506a23fe30fa03033d5c264819613ff20de18bcd962e3e8e234: Status 404 returned error can't find the container with id 0595898a50927506a23fe30fa03033d5c264819613ff20de18bcd962e3e8e234
Nov 03 13:12:11.061202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:11.061171    4546 generic.go:356] "Generic (PLEG): container finished" podID="373ee556-9550-482f-8806-71b0d7147ba1" containerID="0e2d06bf0a64fca1d2cb63593167d09fb133ce395b334c4064ab176c1420fa5a" exitCode=0
Nov 03 13:12:11.061694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:11.061264    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz" event={"ID":"373ee556-9550-482f-8806-71b0d7147ba1","Type":"ContainerDied","Data":"0e2d06bf0a64fca1d2cb63593167d09fb133ce395b334c4064ab176c1420fa5a"}
Nov 03 13:12:11.061694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:11.061315    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz" event={"ID":"373ee556-9550-482f-8806-71b0d7147ba1","Type":"ContainerStarted","Data":"0595898a50927506a23fe30fa03033d5c264819613ff20de18bcd962e3e8e234"}
Nov 03 13:12:12.236598 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.236571    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:12.312730 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.312690    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf\" (UniqueName: \"kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf\") pod \"373ee556-9550-482f-8806-71b0d7147ba1\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") "
Nov 03 13:12:12.312980 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.312966    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca\") pod \"373ee556-9550-482f-8806-71b0d7147ba1\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") "
Nov 03 13:12:12.313053 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.313040    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca\") pod \"373ee556-9550-482f-8806-71b0d7147ba1\" (UID: \"373ee556-9550-482f-8806-71b0d7147ba1\") "
Nov 03 13:12:12.313335 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.313305    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca" pod "373ee556-9550-482f-8806-71b0d7147ba1" (UID: "373ee556-9550-482f-8806-71b0d7147ba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:12:12.313517 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.313499    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:12.315069 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.315047    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf" pod "373ee556-9550-482f-8806-71b0d7147ba1" (UID: "373ee556-9550-482f-8806-71b0d7147ba1"). InnerVolumeSpecName "kube-api-access-lr2mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:12:12.315185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.315119    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca" pod "373ee556-9550-482f-8806-71b0d7147ba1" (UID: "373ee556-9550-482f-8806-71b0d7147ba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:12:12.414848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.414813    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2mf\" (UniqueName: \"kubernetes.io/projected/373ee556-9550-482f-8806-71b0d7147ba1-kube-api-access-lr2mf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:12.414848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:12.414842    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373ee556-9550-482f-8806-71b0d7147ba1-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:13.077249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:13.077218    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"
Nov 03 13:12:13.077249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:13.077237    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz" event={"ID":"373ee556-9550-482f-8806-71b0d7147ba1","Type":"ContainerDied","Data":"0595898a50927506a23fe30fa03033d5c264819613ff20de18bcd962e3e8e234"}
Nov 03 13:12:13.077525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:12:13.077285    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0595898a50927506a23fe30fa03033d5c264819613ff20de18bcd962e3e8e234"
Nov 03 13:12:37.652378 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:12:37.652304    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:13:02.199693 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:02.199661    4546 scope.go:122] "RemoveContainer" containerID="22a8f958765f6ab8dfdaff393dca9d6556980503df6adad01e1e4529c7c3b029"
Nov 03 13:13:07.053178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:07.053139    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"]
Nov 03 13:13:07.098966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:07.098927    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b402cf6f-rz7js"]
Nov 03 13:13:07.652888 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:13:07.652851    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:13:08.573956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:08.573924    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45988ae0-773f-408c-b984-24064217cf8f" path="/var/lib/kubelet/pods/45988ae0-773f-408c-b984-24064217cf8f/volumes"
Nov 03 13:13:16.044018 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:16.043988    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"]
Nov 03 13:13:16.089983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:16.089949    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d16f979-qvjkn"]
Nov 03 13:13:16.573733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:16.573703    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1175b324-73ae-49be-9a14-955267d71f6b" path="/var/lib/kubelet/pods/1175b324-73ae-49be-9a14-955267d71f6b/volumes"
Nov 03 13:13:33.044829 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:33.044743    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"]
Nov 03 13:13:33.091165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:33.091123    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff63217-vv6dh"]
Nov 03 13:13:34.574216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:34.574169    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60d0580-3ec0-48cb-9d8a-aa023c9fea84" path="/var/lib/kubelet/pods/b60d0580-3ec0-48cb-9d8a-aa023c9fea84/volumes"
Nov 03 13:13:37.653445 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:13:37.653407    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:13:44.581962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:44.581888    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"]
Nov 03 13:13:44.696694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:44.696654    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"]
Nov 03 13:13:47.786315 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.786282    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"]
Nov 03 13:13:47.786917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.786900    4546 cpu_manager.go:414] "containerMap: removing container" podUID="373ee556-9550-482f-8806-71b0d7147ba1" containerName="revoke-keys"
Nov 03 13:13:47.786917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.786916    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373ee556-9550-482f-8806-71b0d7147ba1" containerName="revoke-keys"
Nov 03 13:13:47.787062 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.787049    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="373ee556-9550-482f-8806-71b0d7147ba1" containerName="revoke-keys"
Nov 03 13:13:47.790478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.790458    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:47.842101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.842072    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:13:47.888306 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.888217    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:13:47.888637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.888608    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:13:47.888807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.888686    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:13:47.893866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.893842    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"]
Nov 03 13:13:47.928059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.928028    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:47.928233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.928216    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5d2w\" (UniqueName: \"kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:47.928284 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:47.928250    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.032013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.030727    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.032013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.030911    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.032013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.031132    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5d2w\" (UniqueName: \"kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.040868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.035877    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.040868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.036149    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.084202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.084170    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5d2w\" (UniqueName: \"kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w\") pod \"maas-api-revoke-sub-def83461-cm8h4\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.104966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.104934    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:48.486915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.486890    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"]
Nov 03 13:13:48.487656 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:13:48.487635    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81454f3_adfb_47bd_b59c_20bac4481974.slice/crio-2769e7d28fb5dc55a9bbdd016921f0df30420cd4841dad7bd1990174f7b3583f WatchSource:0}: Error finding container 2769e7d28fb5dc55a9bbdd016921f0df30420cd4841dad7bd1990174f7b3583f: Status 404 returned error can't find the container with id 2769e7d28fb5dc55a9bbdd016921f0df30420cd4841dad7bd1990174f7b3583f
Nov 03 13:13:48.723726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:48.723695    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4" event={"ID":"d81454f3-adfb-47bd-b59c-20bac4481974","Type":"ContainerStarted","Data":"2769e7d28fb5dc55a9bbdd016921f0df30420cd4841dad7bd1990174f7b3583f"}
Nov 03 13:13:49.583087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:49.583054    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerName="payload-processing" containerID="cri-o://6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6" gracePeriod=25
Nov 03 13:13:49.697891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:49.697819    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerName="payload-pre-processing" containerID="cri-o://c2656edf247f37b4772fd0f6ed800b497f626c366c99ece96de947b43916d103" gracePeriod=25
Nov 03 13:13:49.731882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:49.731856    4546 generic.go:356] "Generic (PLEG): container finished" podID="d81454f3-adfb-47bd-b59c-20bac4481974" containerID="83a52842e1a7b3d5c3c7f6a09c02e3bf64696537c770b4edd09dede615572c51" exitCode=0
Nov 03 13:13:49.732023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:49.731937    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4" event={"ID":"d81454f3-adfb-47bd-b59c-20bac4481974","Type":"ContainerDied","Data":"83a52842e1a7b3d5c3c7f6a09c02e3bf64696537c770b4edd09dede615572c51"}
Nov 03 13:13:50.384515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:50.384480    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:13:50.898117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:50.898094    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:51.069112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.069080    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca\") pod \"d81454f3-adfb-47bd-b59c-20bac4481974\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") "
Nov 03 13:13:51.069265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.069223    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca\") pod \"d81454f3-adfb-47bd-b59c-20bac4481974\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") "
Nov 03 13:13:51.069325 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.069307    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w\" (UniqueName: \"kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w\") pod \"d81454f3-adfb-47bd-b59c-20bac4481974\" (UID: \"d81454f3-adfb-47bd-b59c-20bac4481974\") "
Nov 03 13:13:51.069476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.069449    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca" pod "d81454f3-adfb-47bd-b59c-20bac4481974" (UID: "d81454f3-adfb-47bd-b59c-20bac4481974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:13:51.069970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.069945    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:51.071452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.071423    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w" pod "d81454f3-adfb-47bd-b59c-20bac4481974" (UID: "d81454f3-adfb-47bd-b59c-20bac4481974"). InnerVolumeSpecName "kube-api-access-g5d2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:51.071555 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.071503    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca" pod "d81454f3-adfb-47bd-b59c-20bac4481974" (UID: "d81454f3-adfb-47bd-b59c-20bac4481974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:13:51.170759 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.170729    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81454f3-adfb-47bd-b59c-20bac4481974-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:51.170880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.170778    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5d2w\" (UniqueName: \"kubernetes.io/projected/d81454f3-adfb-47bd-b59c-20bac4481974-kube-api-access-g5d2w\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:51.745921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.745891    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"
Nov 03 13:13:51.746135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.745926    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4" event={"ID":"d81454f3-adfb-47bd-b59c-20bac4481974","Type":"ContainerDied","Data":"2769e7d28fb5dc55a9bbdd016921f0df30420cd4841dad7bd1990174f7b3583f"}
Nov 03 13:13:51.746135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:51.745972    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2769e7d28fb5dc55a9bbdd016921f0df30420cd4841dad7bd1990174f7b3583f"
Nov 03 13:13:58.520438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.520402    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"]
Nov 03 13:13:58.521420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.521398    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d81454f3-adfb-47bd-b59c-20bac4481974" containerName="revoke-keys"
Nov 03 13:13:58.521491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.521422    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81454f3-adfb-47bd-b59c-20bac4481974" containerName="revoke-keys"
Nov 03 13:13:58.521584 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.521573    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81454f3-adfb-47bd-b59c-20bac4481974" containerName="revoke-keys"
Nov 03 13:13:58.525626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.525604    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.559016 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.558971    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.559162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.559031    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hdn\" (UniqueName: \"kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.559162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.559125    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.577484 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.577453    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"]
Nov 03 13:13:58.618717 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.618687    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:13:58.618921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.618690    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:13:58.618921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.618687    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:13:58.622221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.622198    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:13:58.660421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.660392    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.660548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.660435    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hdn\" (UniqueName: \"kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.660548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.660480    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.661030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.661011    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.709350 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.709327    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:13:58.713258 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.713224    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.757613 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.757583    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:13:58.763450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.763418    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hdn\" (UniqueName: \"kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn\") pod \"maas-api-revoke-sub-d8da864c-ptqc6\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:58.841855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.841783    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"]
Nov 03 13:13:58.842006 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.841985    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy" containerID="cri-o://86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e" gracePeriod=30
Nov 03 13:13:58.846257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.846230    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"]
Nov 03 13:13:58.846448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.846425    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy" containerID="cri-o://39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122" gracePeriod=30
Nov 03 13:13:58.849512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.849487    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"]
Nov 03 13:13:58.849656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.849639    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy" containerID="cri-o://46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14" gracePeriod=30
Nov 03 13:13:58.888131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.888095    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"]
Nov 03 13:13:58.888342 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.888307    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" containerID="cri-o://c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75" gracePeriod=30
Nov 03 13:13:58.889026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.889007    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:13:58.892071 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:58.892048    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:13:59.269277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.269250    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"]
Nov 03 13:13:59.270325 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:13:59.270296    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d359c3_db4d_4592_a4fd_a24453efdb8a.slice/crio-27d0791a1a1ef0b274dc617870315e82b3907ade294aeb9fa7309da2d8bb4f64 WatchSource:0}: Error finding container 27d0791a1a1ef0b274dc617870315e82b3907ade294aeb9fa7309da2d8bb4f64: Status 404 returned error can't find the container with id 27d0791a1a1ef0b274dc617870315e82b3907ade294aeb9fa7309da2d8bb4f64
Nov 03 13:13:59.303578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.303552    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"]
Nov 03 13:13:59.309429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.309407    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.361980 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.361954    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"]
Nov 03 13:13:59.370052 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.370024    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.370165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.370078    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qb4l\" (UniqueName: \"kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.370345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.370320    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.373449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.373424    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:13:59.472031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.472004    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.472157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.472043    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qb4l\" (UniqueName: \"kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.472231 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.472157    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.472608 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.472588    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.474570 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.474551    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.525339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.525320    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qb4l\" (UniqueName: \"kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l\") pod \"maas-api-revoke-sub-e2679896-gz4cv\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.622199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.622175    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:13:59.801115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.801032    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6" event={"ID":"20d359c3-db4d-4592-a4fd-a24453efdb8a","Type":"ContainerStarted","Data":"539a357ea4e2fe76208d9cfa92b0038e10e9ce238624acc5e632abc1ecfe73cc"}
Nov 03 13:13:59.801115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.801087    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6" event={"ID":"20d359c3-db4d-4592-a4fd-a24453efdb8a","Type":"ContainerStarted","Data":"27d0791a1a1ef0b274dc617870315e82b3907ade294aeb9fa7309da2d8bb4f64"}
Nov 03 13:13:59.910695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.910638    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6" podStartSLOduration=1.910624163 podStartE2EDuration="1.910624163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:59.907541034 +0000 UTC m=+4981.479124947" watchObservedRunningTime="2026-11-03 13:13:59.910624163 +0000 UTC m=+4981.482208074"
Nov 03 13:13:59.989186 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:13:59.989151    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"]
Nov 03 13:13:59.989992 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:13:59.989958    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf167a5d_7243_428b_94c8_d08a59a28993.slice/crio-8197b752d6fdf220c3c05f1170fd7b73e0d8c921049c923a5f66cd32c87b4585 WatchSource:0}: Error finding container 8197b752d6fdf220c3c05f1170fd7b73e0d8c921049c923a5f66cd32c87b4585: Status 404 returned error can't find the container with id 8197b752d6fdf220c3c05f1170fd7b73e0d8c921049c923a5f66cd32c87b4585
Nov 03 13:14:00.384651 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:00.384620    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:14:00.809693 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:00.809658    4546 generic.go:356] "Generic (PLEG): container finished" podID="20d359c3-db4d-4592-a4fd-a24453efdb8a" containerID="539a357ea4e2fe76208d9cfa92b0038e10e9ce238624acc5e632abc1ecfe73cc" exitCode=0
Nov 03 13:14:00.810180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:00.809738    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6" event={"ID":"20d359c3-db4d-4592-a4fd-a24453efdb8a","Type":"ContainerDied","Data":"539a357ea4e2fe76208d9cfa92b0038e10e9ce238624acc5e632abc1ecfe73cc"}
Nov 03 13:14:00.811685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:00.811662    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv" event={"ID":"df167a5d-7243-428b-94c8-d08a59a28993","Type":"ContainerStarted","Data":"2a38dc57d9bd33399b67a86e818dd06ec554f213993fb721f9265e81a217ce6d"}
Nov 03 13:14:00.811832 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:00.811699    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv" event={"ID":"df167a5d-7243-428b-94c8-d08a59a28993","Type":"ContainerStarted","Data":"8197b752d6fdf220c3c05f1170fd7b73e0d8c921049c923a5f66cd32c87b4585"}
Nov 03 13:14:01.026092 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:01.026038    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv" podStartSLOduration=2.026025064 podStartE2EDuration="2.026025064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:01.023777313 +0000 UTC m=+4982.595361218" watchObservedRunningTime="2026-11-03 13:14:01.026025064 +0000 UTC m=+4982.597608975"
Nov 03 13:14:01.143042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:01.142948    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dhc85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:14:01.143042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:01.143008    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:14:01.831519 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:01.831483    4546 generic.go:356] "Generic (PLEG): container finished" podID="df167a5d-7243-428b-94c8-d08a59a28993" containerID="2a38dc57d9bd33399b67a86e818dd06ec554f213993fb721f9265e81a217ce6d" exitCode=0
Nov 03 13:14:01.831954 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:01.831575    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv" event={"ID":"df167a5d-7243-428b-94c8-d08a59a28993","Type":"ContainerDied","Data":"2a38dc57d9bd33399b67a86e818dd06ec554f213993fb721f9265e81a217ce6d"}
Nov 03 13:14:01.998955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:01.998930    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:14:02.108193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.108110    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca\") pod \"20d359c3-db4d-4592-a4fd-a24453efdb8a\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") "
Nov 03 13:14:02.108193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.108176    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca\") pod \"20d359c3-db4d-4592-a4fd-a24453efdb8a\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") "
Nov 03 13:14:02.108470 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.108314    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn\" (UniqueName: \"kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn\") pod \"20d359c3-db4d-4592-a4fd-a24453efdb8a\" (UID: \"20d359c3-db4d-4592-a4fd-a24453efdb8a\") "
Nov 03 13:14:02.109157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.109125    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca" pod "20d359c3-db4d-4592-a4fd-a24453efdb8a" (UID: "20d359c3-db4d-4592-a4fd-a24453efdb8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:02.111587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.111556    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn" pod "20d359c3-db4d-4592-a4fd-a24453efdb8a" (UID: "20d359c3-db4d-4592-a4fd-a24453efdb8a"). InnerVolumeSpecName "kube-api-access-k6hdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:02.111701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.111661    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca" pod "20d359c3-db4d-4592-a4fd-a24453efdb8a" (UID: "20d359c3-db4d-4592-a4fd-a24453efdb8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:02.210084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.210047    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:02.210084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.210079    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d359c3-db4d-4592-a4fd-a24453efdb8a-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:02.210084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.210090    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6hdn\" (UniqueName: \"kubernetes.io/projected/20d359c3-db4d-4592-a4fd-a24453efdb8a-kube-api-access-k6hdn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:02.839447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.839360    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"
Nov 03 13:14:02.839447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.839403    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6" event={"ID":"20d359c3-db4d-4592-a4fd-a24453efdb8a","Type":"ContainerDied","Data":"27d0791a1a1ef0b274dc617870315e82b3907ade294aeb9fa7309da2d8bb4f64"}
Nov 03 13:14:02.839987 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:02.839463    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d0791a1a1ef0b274dc617870315e82b3907ade294aeb9fa7309da2d8bb4f64"
Nov 03 13:14:03.000430 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.000404    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:14:03.120878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.120790    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca\") pod \"df167a5d-7243-428b-94c8-d08a59a28993\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") "
Nov 03 13:14:03.120878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.120857    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l\" (UniqueName: \"kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l\") pod \"df167a5d-7243-428b-94c8-d08a59a28993\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") "
Nov 03 13:14:03.121125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.121103    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca\") pod \"df167a5d-7243-428b-94c8-d08a59a28993\" (UID: \"df167a5d-7243-428b-94c8-d08a59a28993\") "
Nov 03 13:14:03.121443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.121416    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca" pod "df167a5d-7243-428b-94c8-d08a59a28993" (UID: "df167a5d-7243-428b-94c8-d08a59a28993"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:03.121733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.121714    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:03.123169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.123143    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca" pod "df167a5d-7243-428b-94c8-d08a59a28993" (UID: "df167a5d-7243-428b-94c8-d08a59a28993"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:03.123525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.123495    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l" pod "df167a5d-7243-428b-94c8-d08a59a28993" (UID: "df167a5d-7243-428b-94c8-d08a59a28993"). InnerVolumeSpecName "kube-api-access-8qb4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:03.223237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.223203    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df167a5d-7243-428b-94c8-d08a59a28993-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:03.223237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.223230    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qb4l\" (UniqueName: \"kubernetes.io/projected/df167a5d-7243-428b-94c8-d08a59a28993-kube-api-access-8qb4l\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:03.847592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.847564    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"
Nov 03 13:14:03.847592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.847574    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv" event={"ID":"df167a5d-7243-428b-94c8-d08a59a28993","Type":"ContainerDied","Data":"8197b752d6fdf220c3c05f1170fd7b73e0d8c921049c923a5f66cd32c87b4585"}
Nov 03 13:14:03.848154 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:03.847622    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8197b752d6fdf220c3c05f1170fd7b73e0d8c921049c923a5f66cd32c87b4585"
Nov 03 13:14:04.196794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.196214    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 13:14:04.236041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.236014    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.236419 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.236399    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.236617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.236591    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.236797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.236782    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.236945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.236931    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.237145 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.237131    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.237259 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.237245    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.237399 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.237386    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.237511 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.237499    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs\") pod \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\" (UID: \"a752bfa7-4293-4c5a-b089-a8c1fd37bd41\") "
Nov 03 13:14:04.238399 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.238378    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.240908 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.240885    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:04.243024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.242136    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.243024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.242397    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.243575 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.243546    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.245210 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.244151    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "kube-api-access-zgjgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.245904 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.245724    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:04.247084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.247059    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.250940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.250916    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token" pod "a752bfa7-4293-4c5a-b089-a8c1fd37bd41" (UID: "a752bfa7-4293-4c5a-b089-a8c1fd37bd41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.282557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.282533    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 13:14:04.285421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.285399    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 13:14:04.299678 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.299652    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 13:14:04.339489 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339456    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.339675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339507    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.339675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339565    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.339675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339596    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.339675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339629    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.339675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339662    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339708    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339778    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339816    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339872    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339881    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339904    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339951    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.339985    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.340017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340018    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340044    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340093    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340168    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340196    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340209    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340245    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340284    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340349    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340412    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340443    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340433    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340476    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340516    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340541    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz\") pod \"ea624674-bd74-4592-9a75-cc59fcaa619f\" (UID: \"ea624674-bd74-4592-9a75-cc59fcaa619f\") "
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340619    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo\") pod \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\" (UID: \"e75b32e5-fec4-4d87-b282-074e1ef9c8bb\") "
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340661    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert\") pod \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\" (UID: \"91662547-69e9-4583-b8d1-5ca79ff8e5c3\") "
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340684    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340704    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340923    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.340956    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:04.341357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.341240    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:04.341777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.341412    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.341777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.341494    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.341777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.341608    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.341961 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.341806    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.344733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.344704    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.345259 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.345234    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.345545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.345521    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.345867 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.345843    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:04.345961 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.345913    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.346250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346225    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:04.346340 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346294    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:04.346810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346785    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346817    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea624674-bd74-4592-9a75-cc59fcaa619f-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346832    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346845    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346858    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346872    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346885    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.346909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346898    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346912    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346926    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346941    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346957    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346970    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346984    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.346997    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347011    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347024    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347038    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347050    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347047    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347065    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgjgn\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-kube-api-access-zgjgn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347079    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347092    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347107    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347125    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea624674-bd74-4592-9a75-cc59fcaa619f-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347138    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347152    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347165    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347179    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.347320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347192    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a752bfa7-4293-4c5a-b089-a8c1fd37bd41-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.348592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347532    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "kube-api-access-d2slb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.348592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.347728    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.348592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.348338    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.348592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.348473    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token" pod "91662547-69e9-4583-b8d1-5ca79ff8e5c3" (UID: "91662547-69e9-4583-b8d1-5ca79ff8e5c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.349028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.349008    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz" pod "ea624674-bd74-4592-9a75-cc59fcaa619f" (UID: "ea624674-bd74-4592-9a75-cc59fcaa619f"). InnerVolumeSpecName "kube-api-access-btnwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.350008 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.349989    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h" pod "e75b32e5-fec4-4d87-b282-074e1ef9c8bb" (UID: "e75b32e5-fec4-4d87-b282-074e1ef9c8bb"). InnerVolumeSpecName "kube-api-access-9rh9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448853    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnwz\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-kube-api-access-btnwz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448881    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448891    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2slb\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-kube-api-access-d2slb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448901    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91662547-69e9-4583-b8d1-5ca79ff8e5c3-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448911    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rh9h\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-kube-api-access-9rh9h\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448927    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75b32e5-fec4-4d87-b282-074e1ef9c8bb-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.448934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.448935    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea624674-bd74-4592-9a75-cc59fcaa619f-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.856128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.856091    4546 generic.go:356] "Generic (PLEG): container finished" podID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerID="46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14" exitCode=0
Nov 03 13:14:04.856607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.856143    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"
Nov 03 13:14:04.856607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.856173    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" event={"ID":"a752bfa7-4293-4c5a-b089-a8c1fd37bd41","Type":"ContainerDied","Data":"46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14"}
Nov 03 13:14:04.856607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.856229    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv" event={"ID":"a752bfa7-4293-4c5a-b089-a8c1fd37bd41","Type":"ContainerDied","Data":"5bd626ba78a45ca016cb2bbe3cda1b8e66e2c8565f8c309b815abaf820651b7d"}
Nov 03 13:14:04.856607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.856259    4546 scope.go:122] "RemoveContainer" containerID="46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14"
Nov 03 13:14:04.857823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.857800    4546 generic.go:356] "Generic (PLEG): container finished" podID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerID="c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75" exitCode=0
Nov 03 13:14:04.857959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.857944    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" event={"ID":"ea624674-bd74-4592-9a75-cc59fcaa619f","Type":"ContainerDied","Data":"c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75"}
Nov 03 13:14:04.858008 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.857980    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85" event={"ID":"ea624674-bd74-4592-9a75-cc59fcaa619f","Type":"ContainerDied","Data":"c53e2c75e1f5c7f05e2fd07cdae8392e5bb05fdd58f6650febd00c299a0111ad"}
Nov 03 13:14:04.858056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.858027    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"
Nov 03 13:14:04.859944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.859922    4546 generic.go:356] "Generic (PLEG): container finished" podID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerID="86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e" exitCode=0
Nov 03 13:14:04.860049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.859976    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"
Nov 03 13:14:04.860049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.860005    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" event={"ID":"91662547-69e9-4583-b8d1-5ca79ff8e5c3","Type":"ContainerDied","Data":"86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e"}
Nov 03 13:14:04.860163 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.860049    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj" event={"ID":"91662547-69e9-4583-b8d1-5ca79ff8e5c3","Type":"ContainerDied","Data":"77a1943f855a5e97f40dc67a8aaa7bf675d6f5a329bd1c5a2be2825200444102"}
Nov 03 13:14:04.861680 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.861651    4546 generic.go:356] "Generic (PLEG): container finished" podID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerID="39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122" exitCode=0
Nov 03 13:14:04.861817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.861682    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" event={"ID":"e75b32e5-fec4-4d87-b282-074e1ef9c8bb","Type":"ContainerDied","Data":"39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122"}
Nov 03 13:14:04.861817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.861705    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"
Nov 03 13:14:04.861817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.861722    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7" event={"ID":"e75b32e5-fec4-4d87-b282-074e1ef9c8bb","Type":"ContainerDied","Data":"6e242e3e6635885aa5ca30ece8a154f64107a6d8f0ff47693994bbe1b7da72fa"}
Nov 03 13:14:04.868709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.868691    4546 scope.go:122] "RemoveContainer" containerID="46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14"
Nov 03 13:14:04.869054 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:04.869030    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14\": container with ID starting with 46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14 not found: ID does not exist" containerID="46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14"
Nov 03 13:14:04.869147 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.869061    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14"} err="failed to get container status \"46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14\": rpc error: code = NotFound desc = could not find container \"46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14\": container with ID starting with 46ffe7a3032076c7bca433e633b706bfdee7fa0aff8e3985fe2dc6438cdaec14 not found: ID does not exist"
Nov 03 13:14:04.869147 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.869088    4546 scope.go:122] "RemoveContainer" containerID="c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75"
Nov 03 13:14:04.882130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.882113    4546 scope.go:122] "RemoveContainer" containerID="c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75"
Nov 03 13:14:04.882400 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:04.882381    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75\": container with ID starting with c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75 not found: ID does not exist" containerID="c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75"
Nov 03 13:14:04.882481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.882409    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75"} err="failed to get container status \"c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75\": rpc error: code = NotFound desc = could not find container \"c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75\": container with ID starting with c72628037bcb9cca0cda9236c0fddadbe915a57016e307d44073a3576487dd75 not found: ID does not exist"
Nov 03 13:14:04.882481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.882425    4546 scope.go:122] "RemoveContainer" containerID="86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e"
Nov 03 13:14:04.891473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.891456    4546 scope.go:122] "RemoveContainer" containerID="86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e"
Nov 03 13:14:04.891720 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:04.891703    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e\": container with ID starting with 86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e not found: ID does not exist" containerID="86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e"
Nov 03 13:14:04.891794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.891728    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e"} err="failed to get container status \"86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e\": rpc error: code = NotFound desc = could not find container \"86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e\": container with ID starting with 86cda62dd1983a4009d0e399ef65e94caad62bbadafa2756720c5a0f7051b62e not found: ID does not exist"
Nov 03 13:14:04.891794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.891761    4546 scope.go:122] "RemoveContainer" containerID="39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122"
Nov 03 13:14:04.900569 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.900554    4546 scope.go:122] "RemoveContainer" containerID="39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122"
Nov 03 13:14:04.900845 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:04.900822    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122\": container with ID starting with 39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122 not found: ID does not exist" containerID="39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122"
Nov 03 13:14:04.900954 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:04.900852    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122"} err="failed to get container status \"39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122\": rpc error: code = NotFound desc = could not find container \"39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122\": container with ID starting with 39e753c171a737dc266c727a3a5770a9fb44a899962d80fd195e9007e080e122 not found: ID does not exist"
Nov 03 13:14:05.013222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.013189    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"]
Nov 03 13:14:05.058964 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.058931    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jzzd7"]
Nov 03 13:14:05.119380 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.119285    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"]
Nov 03 13:14:05.123136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.123110    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dhc85"]
Nov 03 13:14:05.222688 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.222653    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"]
Nov 03 13:14:05.226309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.226274    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7m2kj"]
Nov 03 13:14:05.326766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.326724    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"]
Nov 03 13:14:05.330535 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:05.330504    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xrrqv"]
Nov 03 13:14:06.573696 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:06.573665    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" path="/var/lib/kubelet/pods/91662547-69e9-4583-b8d1-5ca79ff8e5c3/volumes"
Nov 03 13:14:06.574201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:06.574185    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" path="/var/lib/kubelet/pods/a752bfa7-4293-4c5a-b089-a8c1fd37bd41/volumes"
Nov 03 13:14:06.574617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:06.574605    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" path="/var/lib/kubelet/pods/e75b32e5-fec4-4d87-b282-074e1ef9c8bb/volumes"
Nov 03 13:14:06.575358 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:06.575342    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" path="/var/lib/kubelet/pods/ea624674-bd74-4592-9a75-cc59fcaa619f/volumes"
Nov 03 13:14:07.663678 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:07.657958    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:14:09.374214 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.374182    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:14:09.858185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.857785    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:14:09.901477 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.901448    4546 generic.go:356] "Generic (PLEG): container finished" podID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerID="c2656edf247f37b4772fd0f6ed800b497f626c366c99ece96de947b43916d103" exitCode=0
Nov 03 13:14:09.901628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.901524    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" event={"ID":"56f821b8-0b47-4e38-95c8-66bf651ca945","Type":"ContainerDied","Data":"c2656edf247f37b4772fd0f6ed800b497f626c366c99ece96de947b43916d103"}
Nov 03 13:14:09.902999 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.902979    4546 generic.go:356] "Generic (PLEG): container finished" podID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerID="6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6" exitCode=0
Nov 03 13:14:09.903126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.903006    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" event={"ID":"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217","Type":"ContainerDied","Data":"6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6"}
Nov 03 13:14:09.903126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.903025    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"
Nov 03 13:14:09.903126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.903054    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt" event={"ID":"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217","Type":"ContainerDied","Data":"7fb673ca26c670981b78b772b6e423a95aeaff56e9d48e586843765086d25f99"}
Nov 03 13:14:09.903126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.903079    4546 scope.go:122] "RemoveContainer" containerID="6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6"
Nov 03 13:14:09.915526 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.915505    4546 scope.go:122] "RemoveContainer" containerID="6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6"
Nov 03 13:14:09.915887 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:09.915856    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6\": container with ID starting with 6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6 not found: ID does not exist" containerID="6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6"
Nov 03 13:14:09.916013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.915895    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6"} err="failed to get container status \"6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6\": rpc error: code = NotFound desc = could not find container \"6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6\": container with ID starting with 6adb4d8c784cea5d5934f04dfbada58b2addf0516e50c54b27e498370e0041b6 not found: ID does not exist"
Nov 03 13:14:09.957272 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957211    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"]
Nov 03 13:14:09.957891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957874    4546 cpu_manager.go:414] "containerMap: removing container" podUID="df167a5d-7243-428b-94c8-d08a59a28993" containerName="revoke-keys"
Nov 03 13:14:09.957969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957893    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df167a5d-7243-428b-94c8-d08a59a28993" containerName="revoke-keys"
Nov 03 13:14:09.957969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957914    4546 cpu_manager.go:414] "containerMap: removing container" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy"
Nov 03 13:14:09.957969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957919    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy"
Nov 03 13:14:09.957969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957948    4546 cpu_manager.go:414] "containerMap: removing container" podUID="20d359c3-db4d-4592-a4fd-a24453efdb8a" containerName="revoke-keys"
Nov 03 13:14:09.957969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957953    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d359c3-db4d-4592-a4fd-a24453efdb8a" containerName="revoke-keys"
Nov 03 13:14:09.957969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957968    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957973    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957982    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerName="payload-processing"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.957986    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerName="payload-processing"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958008    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958013    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958020    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958024    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958107    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="df167a5d-7243-428b-94c8-d08a59a28993" containerName="revoke-keys"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958115    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="91662547-69e9-4583-b8d1-5ca79ff8e5c3" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958122    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d359c3-db4d-4592-a4fd-a24453efdb8a" containerName="revoke-keys"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958128    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a752bfa7-4293-4c5a-b089-a8c1fd37bd41" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958135    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" containerName="payload-processing"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958144    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea624674-bd74-4592-9a75-cc59fcaa619f" containerName="istio-proxy"
Nov 03 13:14:09.958164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.958151    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75b32e5-fec4-4d87-b282-074e1ef9c8bb" containerName="istio-proxy"
Nov 03 13:14:09.962869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.962851    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:09.978679 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:09.978659    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:14:10.011985 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.011957    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:14:10.012136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.012041    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"]
Nov 03 13:14:10.012136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.012105    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:14:10.012262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.012169    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:14:10.012262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.012224    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:14:10.017061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.017039    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x\" (UniqueName: \"kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x\") pod \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") "
Nov 03 13:14:10.017215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.017093    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume\") pod \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\" (UID: \"d0a06299-4dd7-46ef-bcb8-8bd6f00cd217\") "
Nov 03 13:14:10.017577 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.017548    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume" pod "d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" (UID: "d0a06299-4dd7-46ef-bcb8-8bd6f00cd217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:10.020002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.019974    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x" pod "d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" (UID: "d0a06299-4dd7-46ef-bcb8-8bd6f00cd217"). InnerVolumeSpecName "kube-api-access-nhd9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:10.057671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.057650    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"]
Nov 03 13:14:10.058226 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.058211    4546 cpu_manager.go:414] "containerMap: removing container" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerName="payload-pre-processing"
Nov 03 13:14:10.058294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.058228    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerName="payload-pre-processing"
Nov 03 13:14:10.058340 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.058331    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" containerName="payload-pre-processing"
Nov 03 13:14:10.061811 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.061794    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.115839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.115812    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"]
Nov 03 13:14:10.118954 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.118931    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn\" (UniqueName: \"kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn\") pod \"56f821b8-0b47-4e38-95c8-66bf651ca945\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") "
Nov 03 13:14:10.119057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.118973    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume\") pod \"56f821b8-0b47-4e38-95c8-66bf651ca945\" (UID: \"56f821b8-0b47-4e38-95c8-66bf651ca945\") "
Nov 03 13:14:10.119324 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119304    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume" pod "56f821b8-0b47-4e38-95c8-66bf651ca945" (UID: "56f821b8-0b47-4e38-95c8-66bf651ca945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:10.119385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119370    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xkm\" (UniqueName: \"kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.119441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119421    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.119554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119533    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.119716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119697    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f821b8-0b47-4e38-95c8-66bf651ca945-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:10.119809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119725    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhd9x\" (UniqueName: \"kubernetes.io/projected/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-kube-api-access-nhd9x\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:10.119809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.119740    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:10.121223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.121204    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn" pod "56f821b8-0b47-4e38-95c8-66bf651ca945" (UID: "56f821b8-0b47-4e38-95c8-66bf651ca945"). InnerVolumeSpecName "kube-api-access-fb5jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:10.221086 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221052    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.221243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221141    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.221243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221197    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2b2s\" (UniqueName: \"kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.221243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221237    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.221419 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221277    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.221553 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221519    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xkm\" (UniqueName: \"kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.221832 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.221808    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb5jn\" (UniqueName: \"kubernetes.io/projected/56f821b8-0b47-4e38-95c8-66bf651ca945-kube-api-access-fb5jn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:10.222158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.222133    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.223638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.223618    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.275223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.275200    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xkm\" (UniqueName: \"kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm\") pod \"maas-api-revoke-sub-a237a96c-52l24\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.287615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.287591    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:10.323249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.323210    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2b2s\" (UniqueName: \"kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.323249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.323252    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.323465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.323276    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.324021 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.323995    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.325592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.325575    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.365892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.365859    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"]
Nov 03 13:14:10.369276 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.369251    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5e9ee9-b979c5859-hm7tt"]
Nov 03 13:14:10.377319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.377295    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2b2s\" (UniqueName: \"kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s\") pod \"maas-api-revoke-sub-8775045f-mmzbl\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.574274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.574247    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a06299-4dd7-46ef-bcb8-8bd6f00cd217" path="/var/lib/kubelet/pods/d0a06299-4dd7-46ef-bcb8-8bd6f00cd217/volumes"
Nov 03 13:14:10.655625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.655602    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"]
Nov 03 13:14:10.656697 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:14:10.656674    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc98211_1a38_412b_b4aa_258282063efd.slice/crio-2dd0125033e8e19adabf5380aac33a763bf7cd8cd7b3cce415fc4cab06486a7a WatchSource:0}: Error finding container 2dd0125033e8e19adabf5380aac33a763bf7cd8cd7b3cce415fc4cab06486a7a: Status 404 returned error can't find the container with id 2dd0125033e8e19adabf5380aac33a763bf7cd8cd7b3cce415fc4cab06486a7a
Nov 03 13:14:10.675119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.675097    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:10.912848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.912779    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24" event={"ID":"8bc98211-1a38-412b-b4aa-258282063efd","Type":"ContainerStarted","Data":"2dd0125033e8e19adabf5380aac33a763bf7cd8cd7b3cce415fc4cab06486a7a"}
Nov 03 13:14:10.914107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.914084    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj" event={"ID":"56f821b8-0b47-4e38-95c8-66bf651ca945","Type":"ContainerDied","Data":"bac8a78d5f240bcc65514f5cab58f3d844ce8d1d10c8b6c4ed2374077b81b567"}
Nov 03 13:14:10.914245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.914135    4546 scope.go:122] "RemoveContainer" containerID="c2656edf247f37b4772fd0f6ed800b497f626c366c99ece96de947b43916d103"
Nov 03 13:14:10.914245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:10.914094    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"
Nov 03 13:14:11.046334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.046308    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"]
Nov 03 13:14:11.047307 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:14:11.047282    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5486f5a0_2107_4ab0_9a5b_2cc662dc9ad5.slice/crio-97ab90e4479c69204fbf38683afb8ef389bceff991dad9bf13d466d2377943e5 WatchSource:0}: Error finding container 97ab90e4479c69204fbf38683afb8ef389bceff991dad9bf13d466d2377943e5: Status 404 returned error can't find the container with id 97ab90e4479c69204fbf38683afb8ef389bceff991dad9bf13d466d2377943e5
Nov 03 13:14:11.067243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.067218    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"]
Nov 03 13:14:11.071449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.071427    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5e9ee9-6c8f79bcbc-lrdxj"]
Nov 03 13:14:11.922723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.922694    4546 generic.go:356] "Generic (PLEG): container finished" podID="5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" containerID="4d2dc609ad8231d6f933e66eb78a3b7fac4a24e21e17d194322081503e79a2ad" exitCode=0
Nov 03 13:14:11.923137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.922730    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl" event={"ID":"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5","Type":"ContainerDied","Data":"4d2dc609ad8231d6f933e66eb78a3b7fac4a24e21e17d194322081503e79a2ad"}
Nov 03 13:14:11.923137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.922806    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl" event={"ID":"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5","Type":"ContainerStarted","Data":"97ab90e4479c69204fbf38683afb8ef389bceff991dad9bf13d466d2377943e5"}
Nov 03 13:14:11.924363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.924342    4546 generic.go:356] "Generic (PLEG): container finished" podID="8bc98211-1a38-412b-b4aa-258282063efd" containerID="2888dd93e980a8b0fdc47c8fef795653250f4fe6e293ae29b1be1b7c5a2d9abe" exitCode=0
Nov 03 13:14:11.924477 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:11.924410    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24" event={"ID":"8bc98211-1a38-412b-b4aa-258282063efd","Type":"ContainerDied","Data":"2888dd93e980a8b0fdc47c8fef795653250f4fe6e293ae29b1be1b7c5a2d9abe"}
Nov 03 13:14:12.572975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:12.572942    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f821b8-0b47-4e38-95c8-66bf651ca945" path="/var/lib/kubelet/pods/56f821b8-0b47-4e38-95c8-66bf651ca945/volumes"
Nov 03 13:14:13.090710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.090681    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:13.132125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.132103    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:13.256148 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256121    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca\") pod \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") "
Nov 03 13:14:13.256332 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256178    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca\") pod \"8bc98211-1a38-412b-b4aa-258282063efd\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") "
Nov 03 13:14:13.256392 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256339    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca\") pod \"8bc98211-1a38-412b-b4aa-258282063efd\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") "
Nov 03 13:14:13.256542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256522    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s\" (UniqueName: \"kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s\") pod \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") "
Nov 03 13:14:13.256623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256539    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca" pod "8bc98211-1a38-412b-b4aa-258282063efd" (UID: "8bc98211-1a38-412b-b4aa-258282063efd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:13.256623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256557    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca\") pod \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\" (UID: \"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5\") "
Nov 03 13:14:13.256623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256590    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm\" (UniqueName: \"kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm\") pod \"8bc98211-1a38-412b-b4aa-258282063efd\" (UID: \"8bc98211-1a38-412b-b4aa-258282063efd\") "
Nov 03 13:14:13.257059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.256908    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca" pod "5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" (UID: "5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:13.257398 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.257372    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:13.257484 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.257404    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:13.258624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.258599    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca" pod "5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" (UID: "5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:13.258713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.258672    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca" pod "8bc98211-1a38-412b-b4aa-258282063efd" (UID: "8bc98211-1a38-412b-b4aa-258282063efd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:13.258865 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.258848    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm" pod "8bc98211-1a38-412b-b4aa-258282063efd" (UID: "8bc98211-1a38-412b-b4aa-258282063efd"). InnerVolumeSpecName "kube-api-access-v9xkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:13.258937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.258922    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s" pod "5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" (UID: "5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5"). InnerVolumeSpecName "kube-api-access-z2b2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:13.358992 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.358966    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc98211-1a38-412b-b4aa-258282063efd-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:13.359135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.358997    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2b2s\" (UniqueName: \"kubernetes.io/projected/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-kube-api-access-z2b2s\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:13.359135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.359010    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xkm\" (UniqueName: \"kubernetes.io/projected/8bc98211-1a38-412b-b4aa-258282063efd-kube-api-access-v9xkm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:13.359135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.359019    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:13.942019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.941982    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"
Nov 03 13:14:13.942019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.941998    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl" event={"ID":"5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5","Type":"ContainerDied","Data":"97ab90e4479c69204fbf38683afb8ef389bceff991dad9bf13d466d2377943e5"}
Nov 03 13:14:13.942315 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.942055    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ab90e4479c69204fbf38683afb8ef389bceff991dad9bf13d466d2377943e5"
Nov 03 13:14:13.943518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.943492    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24" event={"ID":"8bc98211-1a38-412b-b4aa-258282063efd","Type":"ContainerDied","Data":"2dd0125033e8e19adabf5380aac33a763bf7cd8cd7b3cce415fc4cab06486a7a"}
Nov 03 13:14:13.943518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.943509    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"
Nov 03 13:14:13.943781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:13.943538    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd0125033e8e19adabf5380aac33a763bf7cd8cd7b3cce415fc4cab06486a7a"
Nov 03 13:14:20.953833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.953798    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"]
Nov 03 13:14:20.954366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.954353    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" containerName="revoke-keys"
Nov 03 13:14:20.954409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.954368    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" containerName="revoke-keys"
Nov 03 13:14:20.954409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.954390    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8bc98211-1a38-412b-b4aa-258282063efd" containerName="revoke-keys"
Nov 03 13:14:20.954409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.954395    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc98211-1a38-412b-b4aa-258282063efd" containerName="revoke-keys"
Nov 03 13:14:20.954507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.954482    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" containerName="revoke-keys"
Nov 03 13:14:20.954507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.954493    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc98211-1a38-412b-b4aa-258282063efd" containerName="revoke-keys"
Nov 03 13:14:20.959328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:20.959311    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.008192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.008119    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:14:21.050223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.050191    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.050369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.050243    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.050369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.050339    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrvn\" (UniqueName: \"kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.054352 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.054327    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:14:21.054481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.054408    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:14:21.054481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.054431    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:14:21.058351 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.058325    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"]
Nov 03 13:14:21.152884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.152851    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.153428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.153399    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.153814 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.153780    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrvn\" (UniqueName: \"kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.159035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.159011    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.162658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.162607    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.207307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.207240    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrvn\" (UniqueName: \"kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn\") pod \"maas-api-revoke-sub-14fceff6-qjpw5\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.273295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.273269    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:21.637415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.637390    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"]
Nov 03 13:14:21.638731 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:14:21.638697    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8809a730_43d4_4ada_8f92_5acb35d72b13.slice/crio-3c95ffa8b9896fef45de45541a99ac6326052db9f854cd28bbeb3443469eb235 WatchSource:0}: Error finding container 3c95ffa8b9896fef45de45541a99ac6326052db9f854cd28bbeb3443469eb235: Status 404 returned error can't find the container with id 3c95ffa8b9896fef45de45541a99ac6326052db9f854cd28bbeb3443469eb235
Nov 03 13:14:21.995248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.995212    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5" event={"ID":"8809a730-43d4-4ada-8f92-5acb35d72b13","Type":"ContainerStarted","Data":"65c83ddebff2980e7e53f499287ef891185abe909a97422b813671f8146fe84a"}
Nov 03 13:14:21.995624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:21.995268    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5" event={"ID":"8809a730-43d4-4ada-8f92-5acb35d72b13","Type":"ContainerStarted","Data":"3c95ffa8b9896fef45de45541a99ac6326052db9f854cd28bbeb3443469eb235"}
Nov 03 13:14:22.100765 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:22.100709    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5" podStartSLOduration=2.100694207 podStartE2EDuration="2.100694207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:22.099572462 +0000 UTC m=+5003.671156387" watchObservedRunningTime="2026-11-03 13:14:22.100694207 +0000 UTC m=+5003.672278120"
Nov 03 13:14:23.003159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:23.003125    4546 generic.go:356] "Generic (PLEG): container finished" podID="8809a730-43d4-4ada-8f92-5acb35d72b13" containerID="65c83ddebff2980e7e53f499287ef891185abe909a97422b813671f8146fe84a" exitCode=0
Nov 03 13:14:23.003576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:23.003202    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5" event={"ID":"8809a730-43d4-4ada-8f92-5acb35d72b13","Type":"ContainerDied","Data":"65c83ddebff2980e7e53f499287ef891185abe909a97422b813671f8146fe84a"}
Nov 03 13:14:24.149520 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.149497    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:24.188945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.188906    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca\") pod \"8809a730-43d4-4ada-8f92-5acb35d72b13\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") "
Nov 03 13:14:24.189092 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.188981    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn\" (UniqueName: \"kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn\") pod \"8809a730-43d4-4ada-8f92-5acb35d72b13\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") "
Nov 03 13:14:24.189178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.189140    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca\") pod \"8809a730-43d4-4ada-8f92-5acb35d72b13\" (UID: \"8809a730-43d4-4ada-8f92-5acb35d72b13\") "
Nov 03 13:14:24.189256 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.189236    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca" pod "8809a730-43d4-4ada-8f92-5acb35d72b13" (UID: "8809a730-43d4-4ada-8f92-5acb35d72b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:24.189826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.189801    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:24.191230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.191202    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn" pod "8809a730-43d4-4ada-8f92-5acb35d72b13" (UID: "8809a730-43d4-4ada-8f92-5acb35d72b13"). InnerVolumeSpecName "kube-api-access-xsrvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:24.191329 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.191309    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca" pod "8809a730-43d4-4ada-8f92-5acb35d72b13" (UID: "8809a730-43d4-4ada-8f92-5acb35d72b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:24.291409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.291319    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8809a730-43d4-4ada-8f92-5acb35d72b13-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:24.291409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:24.291346    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsrvn\" (UniqueName: \"kubernetes.io/projected/8809a730-43d4-4ada-8f92-5acb35d72b13-kube-api-access-xsrvn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:25.017162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:25.017130    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"
Nov 03 13:14:25.017162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:25.017158    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5" event={"ID":"8809a730-43d4-4ada-8f92-5acb35d72b13","Type":"ContainerDied","Data":"3c95ffa8b9896fef45de45541a99ac6326052db9f854cd28bbeb3443469eb235"}
Nov 03 13:14:25.017391 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:14:25.017205    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c95ffa8b9896fef45de45541a99ac6326052db9f854cd28bbeb3443469eb235"
Nov 03 13:14:37.652980 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:14:37.652939    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:15:00.170775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.170721    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"]
Nov 03 13:15:00.171935 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.171911    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8809a730-43d4-4ada-8f92-5acb35d72b13" containerName="revoke-keys"
Nov 03 13:15:00.171935 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.171935    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8809a730-43d4-4ada-8f92-5acb35d72b13" containerName="revoke-keys"
Nov 03 13:15:00.172117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.172057    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8809a730-43d4-4ada-8f92-5acb35d72b13" containerName="revoke-keys"
Nov 03 13:15:00.178137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.178112    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"]
Nov 03 13:15:00.179885 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.179853    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.182192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.182171    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"]
Nov 03 13:15:00.182366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.182276    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.186666 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.186644    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"]
Nov 03 13:15:00.186839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.186817    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.190640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.190617    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.214772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.214734    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"]
Nov 03 13:15:00.219186 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.219167    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.219347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.219327    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"]
Nov 03 13:15:00.223896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.223876    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"]
Nov 03 13:15:00.224114 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.223982    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.229037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.229013    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"]
Nov 03 13:15:00.229481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.229255    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.236683 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.236660    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"]
Nov 03 13:15:00.236822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.236730    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.240645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.240629    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"]
Nov 03 13:15:00.240835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.240819    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.244376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.244357    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.255456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.255434    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:15:00.255564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.255478    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:15:00.255564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.255514    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:15:00.266639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.266615    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"]
Nov 03 13:15:00.309278 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.309243    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"]
Nov 03 13:15:00.313064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.313035    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"]
Nov 03 13:15:00.320011 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.317787    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"]
Nov 03 13:15:00.322425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.322217    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"]
Nov 03 13:15:00.327178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.327153    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"]
Nov 03 13:15:00.334009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.333981    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.334138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334025    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.334138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334058    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.334267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334205    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzng\" (UniqueName: \"kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.334267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334244    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.334385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334290    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.334385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334370    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.334562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334405    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc57z\" (UniqueName: \"kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.334562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334480    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nrj\" (UniqueName: \"kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.334562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334521    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872jf\" (UniqueName: \"kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf\") pod \"maas-api-key-cleanup-29895195-q2j5f\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.334562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334547    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js84b\" (UniqueName: \"kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.334810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334626    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvqj\" (UniqueName: \"kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.334810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334717    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntmt\" (UniqueName: \"kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.334810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334742    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.334810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334794    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.335038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334855    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895195-q2j5f\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.335038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334926    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.335038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334957    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpx5\" (UniqueName: \"kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.335038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.334975    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.335254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.335080    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.366201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.365739    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"]
Nov 03 13:15:00.372034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.372009    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"]
Nov 03 13:15:00.382082 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.382057    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"]
Nov 03 13:15:00.387135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.387111    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"]
Nov 03 13:15:00.436139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436064    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.436296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436124    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.436296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436183    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.436296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436209    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.436296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436248    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzng\" (UniqueName: \"kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.436296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436273    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436299    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436333    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436366    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc57z\" (UniqueName: \"kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436413    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nrj\" (UniqueName: \"kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436439    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872jf\" (UniqueName: \"kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf\") pod \"maas-api-key-cleanup-29895195-q2j5f\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436466    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js84b\" (UniqueName: \"kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.436583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436519    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvqj\" (UniqueName: \"kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436594    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntmt\" (UniqueName: \"kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436626    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436674    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436701    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895195-q2j5f\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436727    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436783    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpx5\" (UniqueName: \"kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436810    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.436901    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437015    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437015    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.437034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437028    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.437708 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437684    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.437708 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437696    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.437881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437726    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.437881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437764    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895195-q2j5f\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.438010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.437994    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.438265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.438248    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.493354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.493315    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzng\" (UniqueName: \"kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.497701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.497337    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntmt\" (UniqueName: \"kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt\") pod \"maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.497701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.497388    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872jf\" (UniqueName: \"kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf\") pod \"maas-api-key-cleanup-29895195-q2j5f\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.499318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.499295    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpx5\" (UniqueName: \"kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.500130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.500108    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.500834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.500809    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc57z\" (UniqueName: \"kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z\") pod \"maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.501394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.501371    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvqj\" (UniqueName: \"kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj\") pod \"maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.501568 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.501527    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js84b\" (UniqueName: \"kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.502420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.502399    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.502900 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.502880    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nrj\" (UniqueName: \"kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.504591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.504571    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:00.515218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.515194    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:00.528122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.527963    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:00.539862 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.539842    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:00.547273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.547076    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:00.557936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.557817    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:00.566994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.565109    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:00.573844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.573635    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:00.584178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.583966    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:00.796193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.795925    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:00.984782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:00.984742    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"]
Nov 03 13:15:00.987645 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:00.987606    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5017aac_1e27_4654_af4f_912ec2f334d0.slice/crio-326df3f6db376bbd0f0f739f327bcce4686aca762769c9e50c217f3272e36f2d WatchSource:0}: Error finding container 326df3f6db376bbd0f0f739f327bcce4686aca762769c9e50c217f3272e36f2d: Status 404 returned error can't find the container with id 326df3f6db376bbd0f0f739f327bcce4686aca762769c9e50c217f3272e36f2d
Nov 03 13:15:01.102406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.102373    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"]
Nov 03 13:15:01.102567 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.102547    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d187689_6a91_4121_a41a_6e7cc5368be2.slice/crio-aca34ae56bbf5622931e6587a778f8a5c4a26f6b87aaf95ca737938ea27d520e WatchSource:0}: Error finding container aca34ae56bbf5622931e6587a778f8a5c4a26f6b87aaf95ca737938ea27d520e: Status 404 returned error can't find the container with id aca34ae56bbf5622931e6587a778f8a5c4a26f6b87aaf95ca737938ea27d520e
Nov 03 13:15:01.125000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.124452    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"]
Nov 03 13:15:01.126412 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.126384    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda313ff7a_0080_4b91_b3c9_074f7ab14a47.slice/crio-98ca836ccda45f1c00e1f1191efe9f822ce2979cfe3918d128acbdaff0631dca WatchSource:0}: Error finding container 98ca836ccda45f1c00e1f1191efe9f822ce2979cfe3918d128acbdaff0631dca: Status 404 returned error can't find the container with id 98ca836ccda45f1c00e1f1191efe9f822ce2979cfe3918d128acbdaff0631dca
Nov 03 13:15:01.127282 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.127180    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144e51e1_4793_4c76_a1d5_c5003a0774a7.slice/crio-ad7d9d6e949757d5938dd3b721830707b1a5367a36bf840787213de02ae66961 WatchSource:0}: Error finding container ad7d9d6e949757d5938dd3b721830707b1a5367a36bf840787213de02ae66961: Status 404 returned error can't find the container with id ad7d9d6e949757d5938dd3b721830707b1a5367a36bf840787213de02ae66961
Nov 03 13:15:01.129049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.129022    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"]
Nov 03 13:15:01.132921 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.132891    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfccd7ee_752b_4989_9100_58fb95133460.slice/crio-1f449b2267618949dbc00de25827afaf3e003b99187746c0afa162bbdfa1c488 WatchSource:0}: Error finding container 1f449b2267618949dbc00de25827afaf3e003b99187746c0afa162bbdfa1c488: Status 404 returned error can't find the container with id 1f449b2267618949dbc00de25827afaf3e003b99187746c0afa162bbdfa1c488
Nov 03 13:15:01.133079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.133057    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"]
Nov 03 13:15:01.147721 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.147698    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"]
Nov 03 13:15:01.149332 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.149309    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d3e58a_3062_4514_a64d_b323b0a48145.slice/crio-7a1a674388c0fb9377b52fdd18f8935d7d436cf99f21196b118e5d99731d8daf WatchSource:0}: Error finding container 7a1a674388c0fb9377b52fdd18f8935d7d436cf99f21196b118e5d99731d8daf: Status 404 returned error can't find the container with id 7a1a674388c0fb9377b52fdd18f8935d7d436cf99f21196b118e5d99731d8daf
Nov 03 13:15:01.194621 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.194597    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"]
Nov 03 13:15:01.194924 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.194841    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d22a434_5281_4346_bea0_09b1fd978ea1.slice/crio-47b74220f90cea35a89bfc6f7c7b688c076903147f835f777d62f768b11ce55a WatchSource:0}: Error finding container 47b74220f90cea35a89bfc6f7c7b688c076903147f835f777d62f768b11ce55a: Status 404 returned error can't find the container with id 47b74220f90cea35a89bfc6f7c7b688c076903147f835f777d62f768b11ce55a
Nov 03 13:15:01.201450 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.201419    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0a31c5_2493_428c_a4e1_2a50fe3eb8e6.slice/crio-ac58062e249d4048a1ac029d420269176a20bec396ae76759d3d2a1dd4debe53 WatchSource:0}: Error finding container ac58062e249d4048a1ac029d420269176a20bec396ae76759d3d2a1dd4debe53: Status 404 returned error can't find the container with id ac58062e249d4048a1ac029d420269176a20bec396ae76759d3d2a1dd4debe53
Nov 03 13:15:01.201575 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.201483    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"]
Nov 03 13:15:01.234390 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.234233    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"]
Nov 03 13:15:01.237813 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.237777    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"]
Nov 03 13:15:01.277515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.277488    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerStarted","Data":"1f449b2267618949dbc00de25827afaf3e003b99187746c0afa162bbdfa1c488"}
Nov 03 13:15:01.278826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.278782    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerStarted","Data":"98ca836ccda45f1c00e1f1191efe9f822ce2979cfe3918d128acbdaff0631dca"}
Nov 03 13:15:01.280014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.279987    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerStarted","Data":"ac58062e249d4048a1ac029d420269176a20bec396ae76759d3d2a1dd4debe53"}
Nov 03 13:15:01.281247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.281226    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerStarted","Data":"47b74220f90cea35a89bfc6f7c7b688c076903147f835f777d62f768b11ce55a"}
Nov 03 13:15:01.282564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.282545    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" event={"ID":"e5017aac-1e27-4654-af4f-912ec2f334d0","Type":"ContainerStarted","Data":"326df3f6db376bbd0f0f739f327bcce4686aca762769c9e50c217f3272e36f2d"}
Nov 03 13:15:01.283856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.283839    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerStarted","Data":"7a1a674388c0fb9377b52fdd18f8935d7d436cf99f21196b118e5d99731d8daf"}
Nov 03 13:15:01.285053 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.285033    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerStarted","Data":"aca34ae56bbf5622931e6587a778f8a5c4a26f6b87aaf95ca737938ea27d520e"}
Nov 03 13:15:01.286175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:01.286153    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" event={"ID":"144e51e1-4793-4c76-a1d5-c5003a0774a7","Type":"ContainerStarted","Data":"ad7d9d6e949757d5938dd3b721830707b1a5367a36bf840787213de02ae66961"}
Nov 03 13:15:01.318137 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:01.318109    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fe4ebc_7d3b_4ce1_9fc6_d9c8a4e09330.slice/crio-61a47fe458d0e1f4f2e35c236ecf556d2d0decab300d374b283166b6bc5f2fcd WatchSource:0}: Error finding container 61a47fe458d0e1f4f2e35c236ecf556d2d0decab300d374b283166b6bc5f2fcd: Status 404 returned error can't find the container with id 61a47fe458d0e1f4f2e35c236ecf556d2d0decab300d374b283166b6bc5f2fcd
Nov 03 13:15:02.236196 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.236165    4546 scope.go:122] "RemoveContainer" containerID="1df03c766795b32b70b0f0e7dccce93dd9e984d25f76130eecd93b391ce16585"
Nov 03 13:15:02.246809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.246788    4546 scope.go:122] "RemoveContainer" containerID="a9c2419ad4d9102ba7b246b7de4e3b9dcb51206e7363b22fffa2d109e3df32a3"
Nov 03 13:15:02.255813 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.255794    4546 scope.go:122] "RemoveContainer" containerID="879f4b96a45f9f5b39381c490fcafd9a3ce59034d6dfda06f6f2f3d0af4db292"
Nov 03 13:15:02.294651 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.294620    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerStarted","Data":"61c4f39fe757455e939814cfcb108b168975efc0b22c631d879708640ebc967a"}
Nov 03 13:15:02.298835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.297250    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerStarted","Data":"2c8a433b7930a5f4e90f08417d44220e36f31338818cf24b65f27b4beda0f54b"}
Nov 03 13:15:02.302092 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.302069    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" event={"ID":"144e51e1-4793-4c76-a1d5-c5003a0774a7","Type":"ContainerStarted","Data":"ef3012d85dca1aecdd5c2e2248eb7c5d6f3f513cd9b7d93b207531003244b545"}
Nov 03 13:15:02.303588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.303564    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerStarted","Data":"520b24459b0c93e623c08a855c0b00f92cf476b6b453a78112e632322dc3a8cc"}
Nov 03 13:15:02.305228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.305200    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerStarted","Data":"4a0fc4379ddfcb5173e54d56f19719f944e599658bbd0ba635fbb1f9473efc24"}
Nov 03 13:15:02.306660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.306615    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerStarted","Data":"d3331ec5a9c9eee5c98cb68e5a7909af84f1d4b4fec7c9ab8943cc815740d16f"}
Nov 03 13:15:02.306725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.306657    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerStarted","Data":"61a47fe458d0e1f4f2e35c236ecf556d2d0decab300d374b283166b6bc5f2fcd"}
Nov 03 13:15:02.312797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.309298    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerStarted","Data":"3cf750863bb6e955241e3f2cb080469079c426cd0b6ddd7f196c57b7ed9814a0"}
Nov 03 13:15:02.315941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.315908    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerStarted","Data":"faccbc028c57e7e6f76e3cc9be37250fa8fb8fcf9f7b2b0856acf96bbd240a31"}
Nov 03 13:15:02.317648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.317625    4546 generic.go:356] "Generic (PLEG): container finished" podID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerID="4fe226f24e5a1a4e81645674e5654c41f95eaca157303795b033dd648129d0e3" exitCode=60
Nov 03 13:15:02.317741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.317706    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" event={"ID":"e5017aac-1e27-4654-af4f-912ec2f334d0","Type":"ContainerDied","Data":"4fe226f24e5a1a4e81645674e5654c41f95eaca157303795b033dd648129d0e3"}
Nov 03 13:15:02.317966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.317946    4546 scope.go:122] "RemoveContainer" containerID="4fe226f24e5a1a4e81645674e5654c41f95eaca157303795b033dd648129d0e3"
Nov 03 13:15:02.319284 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.319262    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerStarted","Data":"1866c670bc7c877d3120bfbb03f0a7d858abf71b160b4920a174b1b4d8683033"}
Nov 03 13:15:02.319360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.319300    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerStarted","Data":"f411587da82953f8d142ff380b0fd0b9dd3257af02a0c0c94dc52efca4703583"}
Nov 03 13:15:02.432890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.432831    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" podStartSLOduration=2.432813247 podStartE2EDuration="2.432813247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:02.427258059 +0000 UTC m=+5043.998841968" watchObservedRunningTime="2026-11-03 13:15:02.432813247 +0000 UTC m=+5044.004397158"
Nov 03 13:15:02.543249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.543200    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" podStartSLOduration=2.543186297 podStartE2EDuration="2.543186297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:02.541148122 +0000 UTC m=+5044.112732053" watchObservedRunningTime="2026-11-03 13:15:02.543186297 +0000 UTC m=+5044.114770209"
Nov 03 13:15:02.651977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.651897    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" podStartSLOduration=2.651882679 podStartE2EDuration="2.651882679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:02.646892093 +0000 UTC m=+5044.218476038" watchObservedRunningTime="2026-11-03 13:15:02.651882679 +0000 UTC m=+5044.223466590"
Nov 03 13:15:02.755868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.755826    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" podStartSLOduration=2.755810683 podStartE2EDuration="2.755810683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:02.752998664 +0000 UTC m=+5044.324582593" watchObservedRunningTime="2026-11-03 13:15:02.755810683 +0000 UTC m=+5044.327394594"
Nov 03 13:15:02.861815 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.861733    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" podStartSLOduration=2.861716802 podStartE2EDuration="2.861716802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:02.860243179 +0000 UTC m=+5044.431827112" watchObservedRunningTime="2026-11-03 13:15:02.861716802 +0000 UTC m=+5044.433300714"
Nov 03 13:15:02.881990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.881959    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"]
Nov 03 13:15:02.882253 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.882215    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" containerID="cri-o://dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071" gracePeriod=30
Nov 03 13:15:02.966834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:02.966789    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" podStartSLOduration=2.966777082 podStartE2EDuration="2.966777082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:02.964207331 +0000 UTC m=+5044.535791252" watchObservedRunningTime="2026-11-03 13:15:02.966777082 +0000 UTC m=+5044.538360983"
Nov 03 13:15:03.079363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.079310    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" podStartSLOduration=3.079298101 podStartE2EDuration="3.079298101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.076319535 +0000 UTC m=+5044.647903462" watchObservedRunningTime="2026-11-03 13:15:03.079298101 +0000 UTC m=+5044.650882011"
Nov 03 13:15:03.182864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.182813    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" podStartSLOduration=3.182799028 podStartE2EDuration="3.182799028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.180104563 +0000 UTC m=+5044.751688492" watchObservedRunningTime="2026-11-03 13:15:03.182799028 +0000 UTC m=+5044.754382939"
Nov 03 13:15:03.290087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.289963    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" podStartSLOduration=3.289946407 podStartE2EDuration="3.289946407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:03.287531956 +0000 UTC m=+5044.859115891" watchObservedRunningTime="2026-11-03 13:15:03.289946407 +0000 UTC m=+5044.861530321"
Nov 03 13:15:03.327616 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.327583    4546 generic.go:356] "Generic (PLEG): container finished" podID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerID="61c4f39fe757455e939814cfcb108b168975efc0b22c631d879708640ebc967a" exitCode=60
Nov 03 13:15:03.327805 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.327669    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerDied","Data":"61c4f39fe757455e939814cfcb108b168975efc0b22c631d879708640ebc967a"}
Nov 03 13:15:03.327991 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.327976    4546 scope.go:122] "RemoveContainer" containerID="61c4f39fe757455e939814cfcb108b168975efc0b22c631d879708640ebc967a"
Nov 03 13:15:03.329494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.329472    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerID="2c8a433b7930a5f4e90f08417d44220e36f31338818cf24b65f27b4beda0f54b" exitCode=60
Nov 03 13:15:03.329637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.329530    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerDied","Data":"2c8a433b7930a5f4e90f08417d44220e36f31338818cf24b65f27b4beda0f54b"}
Nov 03 13:15:03.329827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.329792    4546 scope.go:122] "RemoveContainer" containerID="2c8a433b7930a5f4e90f08417d44220e36f31338818cf24b65f27b4beda0f54b"
Nov 03 13:15:03.331275 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.331258    4546 generic.go:356] "Generic (PLEG): container finished" podID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerID="ef3012d85dca1aecdd5c2e2248eb7c5d6f3f513cd9b7d93b207531003244b545" exitCode=60
Nov 03 13:15:03.331450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.331384    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" event={"ID":"144e51e1-4793-4c76-a1d5-c5003a0774a7","Type":"ContainerDied","Data":"ef3012d85dca1aecdd5c2e2248eb7c5d6f3f513cd9b7d93b207531003244b545"}
Nov 03 13:15:03.331691 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.331567    4546 scope.go:122] "RemoveContainer" containerID="ef3012d85dca1aecdd5c2e2248eb7c5d6f3f513cd9b7d93b207531003244b545"
Nov 03 13:15:03.333313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.333295    4546 generic.go:356] "Generic (PLEG): container finished" podID="bfccd7ee-752b-4989-9100-58fb95133460" containerID="520b24459b0c93e623c08a855c0b00f92cf476b6b453a78112e632322dc3a8cc" exitCode=60
Nov 03 13:15:03.333393 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.333371    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerDied","Data":"520b24459b0c93e623c08a855c0b00f92cf476b6b453a78112e632322dc3a8cc"}
Nov 03 13:15:03.333583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.333567    4546 scope.go:122] "RemoveContainer" containerID="520b24459b0c93e623c08a855c0b00f92cf476b6b453a78112e632322dc3a8cc"
Nov 03 13:15:03.335001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.334983    4546 generic.go:356] "Generic (PLEG): container finished" podID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerID="4a0fc4379ddfcb5173e54d56f19719f944e599658bbd0ba635fbb1f9473efc24" exitCode=60
Nov 03 13:15:03.335112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.335047    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerDied","Data":"4a0fc4379ddfcb5173e54d56f19719f944e599658bbd0ba635fbb1f9473efc24"}
Nov 03 13:15:03.335249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.335233    4546 scope.go:122] "RemoveContainer" containerID="4a0fc4379ddfcb5173e54d56f19719f944e599658bbd0ba635fbb1f9473efc24"
Nov 03 13:15:03.336587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.336567    4546 generic.go:356] "Generic (PLEG): container finished" podID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerID="d3331ec5a9c9eee5c98cb68e5a7909af84f1d4b4fec7c9ab8943cc815740d16f" exitCode=60
Nov 03 13:15:03.336673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.336638    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerDied","Data":"d3331ec5a9c9eee5c98cb68e5a7909af84f1d4b4fec7c9ab8943cc815740d16f"}
Nov 03 13:15:03.336828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.336814    4546 scope.go:122] "RemoveContainer" containerID="d3331ec5a9c9eee5c98cb68e5a7909af84f1d4b4fec7c9ab8943cc815740d16f"
Nov 03 13:15:03.338286 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.338270    4546 generic.go:356] "Generic (PLEG): container finished" podID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerID="3cf750863bb6e955241e3f2cb080469079c426cd0b6ddd7f196c57b7ed9814a0" exitCode=60
Nov 03 13:15:03.338357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.338334    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerDied","Data":"3cf750863bb6e955241e3f2cb080469079c426cd0b6ddd7f196c57b7ed9814a0"}
Nov 03 13:15:03.338578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.338562    4546 scope.go:122] "RemoveContainer" containerID="3cf750863bb6e955241e3f2cb080469079c426cd0b6ddd7f196c57b7ed9814a0"
Nov 03 13:15:03.340216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.340193    4546 generic.go:356] "Generic (PLEG): container finished" podID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerID="faccbc028c57e7e6f76e3cc9be37250fa8fb8fcf9f7b2b0856acf96bbd240a31" exitCode=60
Nov 03 13:15:03.340493 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.340310    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerDied","Data":"faccbc028c57e7e6f76e3cc9be37250fa8fb8fcf9f7b2b0856acf96bbd240a31"}
Nov 03 13:15:03.340493 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.340445    4546 scope.go:122] "RemoveContainer" containerID="faccbc028c57e7e6f76e3cc9be37250fa8fb8fcf9f7b2b0856acf96bbd240a31"
Nov 03 13:15:03.342168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.341983    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" event={"ID":"e5017aac-1e27-4654-af4f-912ec2f334d0","Type":"ContainerStarted","Data":"4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99"}
Nov 03 13:15:03.343433 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.343414    4546 generic.go:356] "Generic (PLEG): container finished" podID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerID="1866c670bc7c877d3120bfbb03f0a7d858abf71b160b4920a174b1b4d8683033" exitCode=60
Nov 03 13:15:03.343532 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.343457    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerDied","Data":"1866c670bc7c877d3120bfbb03f0a7d858abf71b160b4920a174b1b4d8683033"}
Nov 03 13:15:03.343729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.343706    4546 scope.go:122] "RemoveContainer" containerID="1866c670bc7c877d3120bfbb03f0a7d858abf71b160b4920a174b1b4d8683033"
Nov 03 13:15:03.376231 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.376203    4546 patch_prober.go:29] interesting pod/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:15:03.376432 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:03.376399    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:15:04.352989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.352936    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerStarted","Data":"6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d"}
Nov 03 13:15:04.354806 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.354780    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerStarted","Data":"fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7"}
Nov 03 13:15:04.356595 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.356573    4546 generic.go:356] "Generic (PLEG): container finished" podID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerID="c1742be77004db139b66e0f2790f52a091b5d164f878427ba3bbc5565f570e7a" exitCode=60
Nov 03 13:15:04.356685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.356608    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" event={"ID":"144e51e1-4793-4c76-a1d5-c5003a0774a7","Type":"ContainerDied","Data":"c1742be77004db139b66e0f2790f52a091b5d164f878427ba3bbc5565f570e7a"}
Nov 03 13:15:04.356685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.356677    4546 scope.go:122] "RemoveContainer" containerID="ef3012d85dca1aecdd5c2e2248eb7c5d6f3f513cd9b7d93b207531003244b545"
Nov 03 13:15:04.356983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.356946    4546 scope.go:122] "RemoveContainer" containerID="c1742be77004db139b66e0f2790f52a091b5d164f878427ba3bbc5565f570e7a"
Nov 03 13:15:04.357315 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:04.357291    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm_odh-ai-gateway-infra(144e51e1-4793-4c76-a1d5-c5003a0774a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7"
Nov 03 13:15:04.358673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.358654    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerStarted","Data":"125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55"}
Nov 03 13:15:04.360500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.360461    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerStarted","Data":"abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84"}
Nov 03 13:15:04.362300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.362278    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerStarted","Data":"f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3"}
Nov 03 13:15:04.364257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.364238    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerStarted","Data":"6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459"}
Nov 03 13:15:04.366323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.366298    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerStarted","Data":"abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69"}
Nov 03 13:15:04.368265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.368242    4546 generic.go:356] "Generic (PLEG): container finished" podID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerID="4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99" exitCode=60
Nov 03 13:15:04.368388 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.368356    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" event={"ID":"e5017aac-1e27-4654-af4f-912ec2f334d0","Type":"ContainerDied","Data":"4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99"}
Nov 03 13:15:04.368559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.368538    4546 scope.go:122] "RemoveContainer" containerID="4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99"
Nov 03 13:15:04.368831 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:04.368791    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww_odh-ai-gateway-infra(e5017aac-1e27-4654-af4f-912ec2f334d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0"
Nov 03 13:15:04.370391 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.370371    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerStarted","Data":"31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a"}
Nov 03 13:15:04.371273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.371241    4546 scope.go:122] "RemoveContainer" containerID="4fe226f24e5a1a4e81645674e5654c41f95eaca157303795b033dd648129d0e3"
Nov 03 13:15:04.583780 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:04.583680    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" podStartSLOduration=4.583666141 podStartE2EDuration="4.583666141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:04.581549907 +0000 UTC m=+5046.153133838" watchObservedRunningTime="2026-11-03 13:15:04.583666141 +0000 UTC m=+5046.155250051"
Nov 03 13:15:05.379437 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.379403    4546 scope.go:122] "RemoveContainer" containerID="4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99"
Nov 03 13:15:05.379923 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.379834    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww_odh-ai-gateway-infra(e5017aac-1e27-4654-af4f-912ec2f334d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0"
Nov 03 13:15:05.381009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.380976    4546 generic.go:356] "Generic (PLEG): container finished" podID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerID="31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a" exitCode=60
Nov 03 13:15:05.381169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.381060    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerDied","Data":"31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a"}
Nov 03 13:15:05.381169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.381150    4546 scope.go:122] "RemoveContainer" containerID="1866c670bc7c877d3120bfbb03f0a7d858abf71b160b4920a174b1b4d8683033"
Nov 03 13:15:05.381343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.381324    4546 scope.go:122] "RemoveContainer" containerID="31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a"
Nov 03 13:15:05.381651 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.381623    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2_odh-ai-gateway-infra(e61a57bb-d311-423f-85ba-aead972ea4fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb"
Nov 03 13:15:05.383155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.383138    4546 generic.go:356] "Generic (PLEG): container finished" podID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerID="6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d" exitCode=60
Nov 03 13:15:05.383239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.383217    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerDied","Data":"6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d"}
Nov 03 13:15:05.383415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.383400    4546 scope.go:122] "RemoveContainer" containerID="6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d"
Nov 03 13:15:05.383684 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.383665    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd_odh-ai-gateway-infra(b3d3e58a-3062-4514-a64d-b323b0a48145)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145"
Nov 03 13:15:05.385099 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.385079    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerID="fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7" exitCode=60
Nov 03 13:15:05.385190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.385164    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerDied","Data":"fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7"}
Nov 03 13:15:05.385341 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.385325    4546 scope.go:122] "RemoveContainer" containerID="fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7"
Nov 03 13:15:05.385592 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.385565    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg_odh-ai-gateway-infra(6d187689-6a91-4121-a41a-6e7cc5368be2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2"
Nov 03 13:15:05.387067 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.387051    4546 scope.go:122] "RemoveContainer" containerID="c1742be77004db139b66e0f2790f52a091b5d164f878427ba3bbc5565f570e7a"
Nov 03 13:15:05.387301 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.387279    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm_odh-ai-gateway-infra(144e51e1-4793-4c76-a1d5-c5003a0774a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7"
Nov 03 13:15:05.388680 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.388658    4546 generic.go:356] "Generic (PLEG): container finished" podID="bfccd7ee-752b-4989-9100-58fb95133460" containerID="125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55" exitCode=60
Nov 03 13:15:05.388800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.388726    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerDied","Data":"125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55"}
Nov 03 13:15:05.388935 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.388921    4546 scope.go:122] "RemoveContainer" containerID="125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55"
Nov 03 13:15:05.389208 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.389184    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj_odh-ai-gateway-infra(bfccd7ee-752b-4989-9100-58fb95133460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" podUID="bfccd7ee-752b-4989-9100-58fb95133460"
Nov 03 13:15:05.390662 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.390645    4546 generic.go:356] "Generic (PLEG): container finished" podID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerID="abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84" exitCode=60
Nov 03 13:15:05.390736 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.390718    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerDied","Data":"abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84"}
Nov 03 13:15:05.390928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.390913    4546 scope.go:122] "RemoveContainer" containerID="abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84"
Nov 03 13:15:05.391171 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.391153    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895195-q2j5f_odh-ai-gateway-infra(a313ff7a-0080-4b91-b3c9-074f7ab14a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47"
Nov 03 13:15:05.392526 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.392508    4546 generic.go:356] "Generic (PLEG): container finished" podID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerID="f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3" exitCode=60
Nov 03 13:15:05.392617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.392577    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerDied","Data":"f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3"}
Nov 03 13:15:05.392819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.392800    4546 scope.go:122] "RemoveContainer" containerID="f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3"
Nov 03 13:15:05.393119 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.393097    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv_odh-ai-gateway-infra(33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330"
Nov 03 13:15:05.394587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.394567    4546 generic.go:356] "Generic (PLEG): container finished" podID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerID="6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459" exitCode=60
Nov 03 13:15:05.394683 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.394643    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerDied","Data":"6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459"}
Nov 03 13:15:05.394880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.394867    4546 scope.go:122] "RemoveContainer" containerID="6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459"
Nov 03 13:15:05.395053 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.395036    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk_odh-ai-gateway-infra(fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6"
Nov 03 13:15:05.395902 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.395881    4546 scope.go:122] "RemoveContainer" containerID="61c4f39fe757455e939814cfcb108b168975efc0b22c631d879708640ebc967a"
Nov 03 13:15:05.397047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.397029    4546 generic.go:356] "Generic (PLEG): container finished" podID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerID="abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69" exitCode=60
Nov 03 13:15:05.397131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.397063    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerDied","Data":"abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69"}
Nov 03 13:15:05.397266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.397251    4546 scope.go:122] "RemoveContainer" containerID="abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69"
Nov 03 13:15:05.397446 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:05.397430    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt_odh-ai-gateway-infra(0d22a434-5281-4346-bea0-09b1fd978ea1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1"
Nov 03 13:15:05.411268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.411247    4546 scope.go:122] "RemoveContainer" containerID="2c8a433b7930a5f4e90f08417d44220e36f31338818cf24b65f27b4beda0f54b"
Nov 03 13:15:05.427624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.427602    4546 scope.go:122] "RemoveContainer" containerID="520b24459b0c93e623c08a855c0b00f92cf476b6b453a78112e632322dc3a8cc"
Nov 03 13:15:05.442478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.442458    4546 scope.go:122] "RemoveContainer" containerID="4a0fc4379ddfcb5173e54d56f19719f944e599658bbd0ba635fbb1f9473efc24"
Nov 03 13:15:05.456058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.456037    4546 scope.go:122] "RemoveContainer" containerID="d3331ec5a9c9eee5c98cb68e5a7909af84f1d4b4fec7c9ab8943cc815740d16f"
Nov 03 13:15:05.470160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.470141    4546 scope.go:122] "RemoveContainer" containerID="3cf750863bb6e955241e3f2cb080469079c426cd0b6ddd7f196c57b7ed9814a0"
Nov 03 13:15:05.483554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:05.483535    4546 scope.go:122] "RemoveContainer" containerID="faccbc028c57e7e6f76e3cc9be37250fa8fb8fcf9f7b2b0856acf96bbd240a31"
Nov 03 13:15:06.406216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.406190    4546 scope.go:122] "RemoveContainer" containerID="6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d"
Nov 03 13:15:06.406663 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.406434    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd_odh-ai-gateway-infra(b3d3e58a-3062-4514-a64d-b323b0a48145)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145"
Nov 03 13:15:06.408042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.408021    4546 scope.go:122] "RemoveContainer" containerID="fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7"
Nov 03 13:15:06.408270 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.408249    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg_odh-ai-gateway-infra(6d187689-6a91-4121-a41a-6e7cc5368be2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2"
Nov 03 13:15:06.409786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.409762    4546 scope.go:122] "RemoveContainer" containerID="125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55"
Nov 03 13:15:06.410062 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.410013    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj_odh-ai-gateway-infra(bfccd7ee-752b-4989-9100-58fb95133460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" podUID="bfccd7ee-752b-4989-9100-58fb95133460"
Nov 03 13:15:06.411447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.411427    4546 scope.go:122] "RemoveContainer" containerID="abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84"
Nov 03 13:15:06.411650 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.411627    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895195-q2j5f_odh-ai-gateway-infra(a313ff7a-0080-4b91-b3c9-074f7ab14a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47"
Nov 03 13:15:06.413076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.413053    4546 scope.go:122] "RemoveContainer" containerID="f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3"
Nov 03 13:15:06.413305 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.413281    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv_odh-ai-gateway-infra(33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330"
Nov 03 13:15:06.414636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.414617    4546 scope.go:122] "RemoveContainer" containerID="6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459"
Nov 03 13:15:06.414827 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.414806    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk_odh-ai-gateway-infra(fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6"
Nov 03 13:15:06.416641 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.416623    4546 scope.go:122] "RemoveContainer" containerID="abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69"
Nov 03 13:15:06.416855 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.416837    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt_odh-ai-gateway-infra(0d22a434-5281-4346-bea0-09b1fd978ea1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1"
Nov 03 13:15:06.417958 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:06.417940    4546 scope.go:122] "RemoveContainer" containerID="31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a"
Nov 03 13:15:06.418144 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:06.418128    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2_odh-ai-gateway-infra(e61a57bb-d311-423f-85ba-aead972ea4fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb"
Nov 03 13:15:07.652135 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:07.652102    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:15:08.170017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.169945    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:15:08.345640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345608    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.345850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345670    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.345850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345730    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.345850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345796    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.345978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345891    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.345978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345900    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:08.345978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.345928    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.346117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346012    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:08.346117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346067    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.346233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346132    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.346233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346161    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert\") pod \"213e49b5-465a-480e-955e-c6b83f42194b\" (UID: \"213e49b5-465a-480e-955e-c6b83f42194b\") "
Nov 03 13:15:08.346344 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346287    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:08.346572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346416    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:08.346697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346679    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.346807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346698    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.346807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346710    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.346807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346722    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.346990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.346842    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:08.348317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.348286    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "kube-api-access-gn6dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:08.348423 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.348311    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:08.348423 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.348339    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:15:08.348558 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.348544    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy" pod "213e49b5-465a-480e-955e-c6b83f42194b" (UID: "213e49b5-465a-480e-955e-c6b83f42194b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:08.433465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.433397    4546 generic.go:356] "Generic (PLEG): container finished" podID="213e49b5-465a-480e-955e-c6b83f42194b" containerID="dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071" exitCode=0
Nov 03 13:15:08.433465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.433453    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"
Nov 03 13:15:08.433635 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.433487    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" event={"ID":"213e49b5-465a-480e-955e-c6b83f42194b","Type":"ContainerDied","Data":"dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071"}
Nov 03 13:15:08.433635 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.433577    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87" event={"ID":"213e49b5-465a-480e-955e-c6b83f42194b","Type":"ContainerDied","Data":"524451c71e6acbc4696ae5c537c91264404ee67fa9fe7dec6d3de998b7dcb484"}
Nov 03 13:15:08.433635 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.433606    4546 scope.go:122] "RemoveContainer" containerID="dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071"
Nov 03 13:15:08.444658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.444641    4546 scope.go:122] "RemoveContainer" containerID="ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"
Nov 03 13:15:08.448218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.448201    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.448305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.448222    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6dh\" (UniqueName: \"kubernetes.io/projected/213e49b5-465a-480e-955e-c6b83f42194b-kube-api-access-gn6dh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.448305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.448234    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e49b5-465a-480e-955e-c6b83f42194b-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.448305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.448243    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e49b5-465a-480e-955e-c6b83f42194b-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.448305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.448251    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e49b5-465a-480e-955e-c6b83f42194b-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:08.453922 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.453906    4546 scope.go:122] "RemoveContainer" containerID="dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071"
Nov 03 13:15:08.454211 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:08.454183    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071\": container with ID starting with dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071 not found: ID does not exist" containerID="dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071"
Nov 03 13:15:08.454323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.454222    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071"} err="failed to get container status \"dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071\": rpc error: code = NotFound desc = could not find container \"dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071\": container with ID starting with dcc2e4e993c07436c81cc628ba5461b27ea06b71c0c9078642b99f423e55e071 not found: ID does not exist"
Nov 03 13:15:08.454323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.454244    4546 scope.go:122] "RemoveContainer" containerID="ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"
Nov 03 13:15:08.454495 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:08.454479    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d\": container with ID starting with ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d not found: ID does not exist" containerID="ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"
Nov 03 13:15:08.454563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.454504    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d"} err="failed to get container status \"ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d\": rpc error: code = NotFound desc = could not find container \"ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d\": container with ID starting with ad33395bcdd6a8de799c0c3663af5cccabbe72c59bbbf58af9753abf67a4f93d not found: ID does not exist"
Nov 03 13:15:08.593072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.593037    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"]
Nov 03 13:15:08.602473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:08.602444    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5e9ee9-openshift-default-5c57b9b77c-d4t87"]
Nov 03 13:15:10.573468 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:10.573425    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213e49b5-465a-480e-955e-c6b83f42194b" path="/var/lib/kubelet/pods/213e49b5-465a-480e-955e-c6b83f42194b/volumes"
Nov 03 13:15:11.926664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:11.926623    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"]
Nov 03 13:15:15.566306 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:15.566273    4546 scope.go:122] "RemoveContainer" containerID="31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a"
Nov 03 13:15:15.566733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:15.566309    4546 scope.go:122] "RemoveContainer" containerID="fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7"
Nov 03 13:15:15.566733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:15.566471    4546 scope.go:122] "RemoveContainer" containerID="4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99"
Nov 03 13:15:16.491508 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.491484    4546 generic.go:356] "Generic (PLEG): container finished" podID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerID="dcb36328f9adebac32c0d0be9f028f941572f51a2aa1dbb69e2983a315912ec8" exitCode=60
Nov 03 13:15:16.491656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.491555    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerDied","Data":"dcb36328f9adebac32c0d0be9f028f941572f51a2aa1dbb69e2983a315912ec8"}
Nov 03 13:15:16.491656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.491609    4546 scope.go:122] "RemoveContainer" containerID="31f0340ed6ca03ccc4bf2bcedcb76780689129e5ce91d694e3c479d0175ccf3a"
Nov 03 13:15:16.491956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.491941    4546 scope.go:122] "RemoveContainer" containerID="dcb36328f9adebac32c0d0be9f028f941572f51a2aa1dbb69e2983a315912ec8"
Nov 03 13:15:16.492188 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:16.492168    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2_odh-ai-gateway-infra(e61a57bb-d311-423f-85ba-aead972ea4fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb"
Nov 03 13:15:16.494016 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.493996    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerStarted","Data":"801fce7724d88f311a1b4b25e295d065474c8903fec1b6a819e11f065eaddf85"}
Nov 03 13:15:16.496000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.495983    4546 generic.go:356] "Generic (PLEG): container finished" podID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerID="2575143755c43980a8bb84c18d6eb6f61782e51df24360ce7737f07b888707d5" exitCode=60
Nov 03 13:15:16.496110 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.496009    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" event={"ID":"e5017aac-1e27-4654-af4f-912ec2f334d0","Type":"ContainerDied","Data":"2575143755c43980a8bb84c18d6eb6f61782e51df24360ce7737f07b888707d5"}
Nov 03 13:15:16.496324 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.496308    4546 scope.go:122] "RemoveContainer" containerID="2575143755c43980a8bb84c18d6eb6f61782e51df24360ce7737f07b888707d5"
Nov 03 13:15:16.496552 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:16.496532    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww_odh-ai-gateway-infra(e5017aac-1e27-4654-af4f-912ec2f334d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0"
Nov 03 13:15:16.504148 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.504132    4546 scope.go:122] "RemoveContainer" containerID="4eca7c3d472e81f7bc14fe2250d048b71515ee8f44c9b2be37b0d9a3f0880a99"
Nov 03 13:15:16.566253 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.566222    4546 scope.go:122] "RemoveContainer" containerID="6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d"
Nov 03 13:15:16.566505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.566416    4546 scope.go:122] "RemoveContainer" containerID="125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55"
Nov 03 13:15:16.566858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.566542    4546 scope.go:122] "RemoveContainer" containerID="abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69"
Nov 03 13:15:16.569427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:16.567499    4546 scope.go:122] "RemoveContainer" containerID="abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84"
Nov 03 13:15:17.506549 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.506520    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerStarted","Data":"4a1ffefb4d091a5b7ae3a14dddf173fbf9565eedad308c7557940a5efec12658"}
Nov 03 13:15:17.512771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.509197    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerStarted","Data":"3e7c55952382983e9af56f33f3dfe6dfadb6e486e305e0fa47b71c870e18612c"}
Nov 03 13:15:17.512771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.511361    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerStarted","Data":"60c056e97a92aaa40c26767fc95ea9166fda3b4078da89a3d0e2cfb7cbc14ad7"}
Nov 03 13:15:17.518010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.517983    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerStarted","Data":"57146334492f18a408475708aa9ae74319f806868ff2e888e2b9ea597f6194a3"}
Nov 03 13:15:17.520740 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.520715    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerID="801fce7724d88f311a1b4b25e295d065474c8903fec1b6a819e11f065eaddf85" exitCode=60
Nov 03 13:15:17.520872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.520775    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerDied","Data":"801fce7724d88f311a1b4b25e295d065474c8903fec1b6a819e11f065eaddf85"}
Nov 03 13:15:17.520872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.520827    4546 scope.go:122] "RemoveContainer" containerID="fe3f7d716f4633452269277ad6db2334cd6c6772d4d717a15474c6c226e187c7"
Nov 03 13:15:17.521155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.521136    4546 scope.go:122] "RemoveContainer" containerID="801fce7724d88f311a1b4b25e295d065474c8903fec1b6a819e11f065eaddf85"
Nov 03 13:15:17.521433 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:17.521409    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg_odh-ai-gateway-infra(6d187689-6a91-4121-a41a-6e7cc5368be2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2"
Nov 03 13:15:17.565993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.565964    4546 scope.go:122] "RemoveContainer" containerID="f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3"
Nov 03 13:15:17.625510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.625482    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"]
Nov 03 13:15:17.732626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.732296    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"]
Nov 03 13:15:17.840638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.840607    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"]
Nov 03 13:15:17.903961 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.903941    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:17.907144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.907125    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:17.963171 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963149    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca\") pod \"e5017aac-1e27-4654-af4f-912ec2f334d0\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") "
Nov 03 13:15:17.963285 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963263    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5\" (UniqueName: \"kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5\") pod \"e61a57bb-d311-423f-85ba-aead972ea4fb\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") "
Nov 03 13:15:17.963328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963286    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng\" (UniqueName: \"kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng\") pod \"e5017aac-1e27-4654-af4f-912ec2f334d0\" (UID: \"e5017aac-1e27-4654-af4f-912ec2f334d0\") "
Nov 03 13:15:17.963381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963354    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca\") pod \"e61a57bb-d311-423f-85ba-aead972ea4fb\" (UID: \"e61a57bb-d311-423f-85ba-aead972ea4fb\") "
Nov 03 13:15:17.963496 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963476    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca" pod "e5017aac-1e27-4654-af4f-912ec2f334d0" (UID: "e5017aac-1e27-4654-af4f-912ec2f334d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:17.963796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963770    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca" pod "e61a57bb-d311-423f-85ba-aead972ea4fb" (UID: "e61a57bb-d311-423f-85ba-aead972ea4fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:17.963906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.963859    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5017aac-1e27-4654-af4f-912ec2f334d0-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:17.965377 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.965354    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5" pod "e61a57bb-d311-423f-85ba-aead972ea4fb" (UID: "e61a57bb-d311-423f-85ba-aead972ea4fb"). InnerVolumeSpecName "kube-api-access-rlpx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:17.965464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:17.965416    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng" pod "e5017aac-1e27-4654-af4f-912ec2f334d0" (UID: "e5017aac-1e27-4654-af4f-912ec2f334d0"). InnerVolumeSpecName "kube-api-access-xtzng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:18.065429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.065358    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlpx5\" (UniqueName: \"kubernetes.io/projected/e61a57bb-d311-423f-85ba-aead972ea4fb-kube-api-access-rlpx5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.065429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.065385    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzng\" (UniqueName: \"kubernetes.io/projected/e5017aac-1e27-4654-af4f-912ec2f334d0-kube-api-access-xtzng\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.065429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.065398    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e61a57bb-d311-423f-85ba-aead972ea4fb-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.531026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.531001    4546 generic.go:356] "Generic (PLEG): container finished" podID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerID="2244ee257cc2a26fb93a4973315f40d215c79ae7796d6f586c0c42fdcf0976f9" exitCode=60
Nov 03 13:15:18.531176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.531072    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerDied","Data":"2244ee257cc2a26fb93a4973315f40d215c79ae7796d6f586c0c42fdcf0976f9"}
Nov 03 13:15:18.531176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.531151    4546 scope.go:122] "RemoveContainer" containerID="f097f8957b834758f0d83401c6142a3895aa933db056ae87d9d67b0b163612c3"
Nov 03 13:15:18.531538 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.531513    4546 scope.go:122] "RemoveContainer" containerID="2244ee257cc2a26fb93a4973315f40d215c79ae7796d6f586c0c42fdcf0976f9"
Nov 03 13:15:18.531929 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:18.531902    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv_odh-ai-gateway-infra(33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330"
Nov 03 13:15:18.533017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.532997    4546 generic.go:356] "Generic (PLEG): container finished" podID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerID="60c056e97a92aaa40c26767fc95ea9166fda3b4078da89a3d0e2cfb7cbc14ad7" exitCode=60
Nov 03 13:15:18.533129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.533068    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerDied","Data":"60c056e97a92aaa40c26767fc95ea9166fda3b4078da89a3d0e2cfb7cbc14ad7"}
Nov 03 13:15:18.533435 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.533417    4546 scope.go:122] "RemoveContainer" containerID="60c056e97a92aaa40c26767fc95ea9166fda3b4078da89a3d0e2cfb7cbc14ad7"
Nov 03 13:15:18.533664 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:18.533646    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt_odh-ai-gateway-infra(0d22a434-5281-4346-bea0-09b1fd978ea1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1"
Nov 03 13:15:18.534372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.534356    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"
Nov 03 13:15:18.534372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.534365    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww" event={"ID":"e5017aac-1e27-4654-af4f-912ec2f334d0","Type":"ContainerDied","Data":"326df3f6db376bbd0f0f739f327bcce4686aca762769c9e50c217f3272e36f2d"}
Nov 03 13:15:18.535737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.535707    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"
Nov 03 13:15:18.535737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.535726    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2" event={"ID":"e61a57bb-d311-423f-85ba-aead972ea4fb","Type":"ContainerDied","Data":"f411587da82953f8d142ff380b0fd0b9dd3257af02a0c0c94dc52efca4703583"}
Nov 03 13:15:18.537823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.537801    4546 generic.go:356] "Generic (PLEG): container finished" podID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerID="57146334492f18a408475708aa9ae74319f806868ff2e888e2b9ea597f6194a3" exitCode=60
Nov 03 13:15:18.537919 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.537885    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerDied","Data":"57146334492f18a408475708aa9ae74319f806868ff2e888e2b9ea597f6194a3"}
Nov 03 13:15:18.538289 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.538268    4546 scope.go:122] "RemoveContainer" containerID="57146334492f18a408475708aa9ae74319f806868ff2e888e2b9ea597f6194a3"
Nov 03 13:15:18.538519 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:18.538496    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd_odh-ai-gateway-infra(b3d3e58a-3062-4514-a64d-b323b0a48145)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145"
Nov 03 13:15:18.541650 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.541626    4546 generic.go:356] "Generic (PLEG): container finished" podID="bfccd7ee-752b-4989-9100-58fb95133460" containerID="4a1ffefb4d091a5b7ae3a14dddf173fbf9565eedad308c7557940a5efec12658" exitCode=60
Nov 03 13:15:18.541804 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.541651    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerDied","Data":"4a1ffefb4d091a5b7ae3a14dddf173fbf9565eedad308c7557940a5efec12658"}
Nov 03 13:15:18.541993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.541974    4546 scope.go:122] "RemoveContainer" containerID="4a1ffefb4d091a5b7ae3a14dddf173fbf9565eedad308c7557940a5efec12658"
Nov 03 13:15:18.542238 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:18.542215    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj_odh-ai-gateway-infra(bfccd7ee-752b-4989-9100-58fb95133460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" podUID="bfccd7ee-752b-4989-9100-58fb95133460"
Nov 03 13:15:18.543590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.543571    4546 generic.go:356] "Generic (PLEG): container finished" podID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerID="3e7c55952382983e9af56f33f3dfe6dfadb6e486e305e0fa47b71c870e18612c" exitCode=60
Nov 03 13:15:18.543677 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.543604    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerDied","Data":"3e7c55952382983e9af56f33f3dfe6dfadb6e486e305e0fa47b71c870e18612c"}
Nov 03 13:15:18.543970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.543956    4546 scope.go:122] "RemoveContainer" containerID="3e7c55952382983e9af56f33f3dfe6dfadb6e486e305e0fa47b71c870e18612c"
Nov 03 13:15:18.544207 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:18.544187    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895195-q2j5f_odh-ai-gateway-infra(a313ff7a-0080-4b91-b3c9-074f7ab14a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47"
Nov 03 13:15:18.564175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.564151    4546 scope.go:122] "RemoveContainer" containerID="abeb685778f621435a2e2bbd1e853dcba541efcc298178b310fcdbd312854f69"
Nov 03 13:15:18.566463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.566444    4546 scope.go:122] "RemoveContainer" containerID="6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459"
Nov 03 13:15:18.566617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.566541    4546 scope.go:122] "RemoveContainer" containerID="c1742be77004db139b66e0f2790f52a091b5d164f878427ba3bbc5565f570e7a"
Nov 03 13:15:18.585672 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.585653    4546 scope.go:122] "RemoveContainer" containerID="2575143755c43980a8bb84c18d6eb6f61782e51df24360ce7737f07b888707d5"
Nov 03 13:15:18.597089 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.597071    4546 scope.go:122] "RemoveContainer" containerID="dcb36328f9adebac32c0d0be9f028f941572f51a2aa1dbb69e2983a315912ec8"
Nov 03 13:15:18.617362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.616889    4546 scope.go:122] "RemoveContainer" containerID="6273082cd1f76dc131c9d9919f84d96190e3ef0e76098207e12230f20137c06d"
Nov 03 13:15:18.633917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.633877    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"]
Nov 03 13:15:18.638668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.638646    4546 scope.go:122] "RemoveContainer" containerID="125913ef608cf782670c889ed148936dec62ae14bdb43226f9662802bd0a0e55"
Nov 03 13:15:18.743438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.743418    4546 scope.go:122] "RemoveContainer" containerID="abc9b39397ca9aefb46eb07bdb01463b8a231832306accb86f31f64edff8df84"
Nov 03 13:15:18.746657 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.746633    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:18.752263 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.752239    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"]
Nov 03 13:15:18.859148 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.859122    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"]
Nov 03 13:15:18.885885 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.885860    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca\") pod \"6d187689-6a91-4121-a41a-6e7cc5368be2\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") "
Nov 03 13:15:18.886108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.886095    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj\" (UniqueName: \"kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj\") pod \"6d187689-6a91-4121-a41a-6e7cc5368be2\" (UID: \"6d187689-6a91-4121-a41a-6e7cc5368be2\") "
Nov 03 13:15:18.886261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.886239    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca" pod "6d187689-6a91-4121-a41a-6e7cc5368be2" (UID: "6d187689-6a91-4121-a41a-6e7cc5368be2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:18.886392 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.886379    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d187689-6a91-4121-a41a-6e7cc5368be2-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:18.888349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.888327    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj" pod "6d187689-6a91-4121-a41a-6e7cc5368be2" (UID: "6d187689-6a91-4121-a41a-6e7cc5368be2"). InnerVolumeSpecName "kube-api-access-84nrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:18.976529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.976501    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"]
Nov 03 13:15:18.981791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.981760    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1250b3-29895195-6q4x2"]
Nov 03 13:15:18.987923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:18.987901    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nrj\" (UniqueName: \"kubernetes.io/projected/6d187689-6a91-4121-a41a-6e7cc5368be2-kube-api-access-84nrj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.088978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.088935    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"]
Nov 03 13:15:19.092775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.092738    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6fb3a8-29895195-dvxww"]
Nov 03 13:15:19.101672 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.101650    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"]
Nov 03 13:15:19.554881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.554853    4546 generic.go:356] "Generic (PLEG): container finished" podID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerID="3cfc20ae39c6bc09c9c2981226ae28fc317d26b016f45c62d9076a13a1f25cbc" exitCode=60
Nov 03 13:15:19.555089 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.554924    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerDied","Data":"3cfc20ae39c6bc09c9c2981226ae28fc317d26b016f45c62d9076a13a1f25cbc"}
Nov 03 13:15:19.555089 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.554980    4546 scope.go:122] "RemoveContainer" containerID="6cfc16fbd30b72789103c67c7859abe01304d2d9b34727ff474b8207f1a8c459"
Nov 03 13:15:19.555424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.555398    4546 scope.go:122] "RemoveContainer" containerID="3cfc20ae39c6bc09c9c2981226ae28fc317d26b016f45c62d9076a13a1f25cbc"
Nov 03 13:15:19.555792 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:19.555765    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk_odh-ai-gateway-infra(fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6"
Nov 03 13:15:19.563936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.563915    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"
Nov 03 13:15:19.564061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.563955    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg" event={"ID":"6d187689-6a91-4121-a41a-6e7cc5368be2","Type":"ContainerDied","Data":"aca34ae56bbf5622931e6587a778f8a5c4a26f6b87aaf95ca737938ea27d520e"}
Nov 03 13:15:19.567167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.567147    4546 generic.go:356] "Generic (PLEG): container finished" podID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerID="07146d9c4110aa67b28fc06d99c09cffa5bb05b26dd59ff438f2fc105d5ff2d0" exitCode=60
Nov 03 13:15:19.567291 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.567218    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" event={"ID":"144e51e1-4793-4c76-a1d5-c5003a0774a7","Type":"ContainerDied","Data":"07146d9c4110aa67b28fc06d99c09cffa5bb05b26dd59ff438f2fc105d5ff2d0"}
Nov 03 13:15:19.567570 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.567549    4546 scope.go:122] "RemoveContainer" containerID="07146d9c4110aa67b28fc06d99c09cffa5bb05b26dd59ff438f2fc105d5ff2d0"
Nov 03 13:15:19.567864 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:19.567838    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm_odh-ai-gateway-infra(144e51e1-4793-4c76-a1d5-c5003a0774a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7"
Nov 03 13:15:19.579294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.579270    4546 scope.go:122] "RemoveContainer" containerID="801fce7724d88f311a1b4b25e295d065474c8903fec1b6a819e11f065eaddf85"
Nov 03 13:15:19.603796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.598614    4546 scope.go:122] "RemoveContainer" containerID="c1742be77004db139b66e0f2790f52a091b5d164f878427ba3bbc5565f570e7a"
Nov 03 13:15:19.663253 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.663174    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"]
Nov 03 13:15:19.771669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.771638    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:19.813786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.809543    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca\") pod \"0d22a434-5281-4346-bea0-09b1fd978ea1\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") "
Nov 03 13:15:19.813786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.809649    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt\" (UniqueName: \"kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt\") pod \"0d22a434-5281-4346-bea0-09b1fd978ea1\" (UID: \"0d22a434-5281-4346-bea0-09b1fd978ea1\") "
Nov 03 13:15:19.813786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.810866    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca" pod "0d22a434-5281-4346-bea0-09b1fd978ea1" (UID: "0d22a434-5281-4346-bea0-09b1fd978ea1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:19.826803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.819166    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt" pod "0d22a434-5281-4346-bea0-09b1fd978ea1" (UID: "0d22a434-5281-4346-bea0-09b1fd978ea1"). InnerVolumeSpecName "kube-api-access-5ntmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:19.882193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.882169    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:19.885912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.885889    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:19.900506 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.900474    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:19.902618 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.902598    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:19.911599 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911570    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca\") pod \"b3d3e58a-3062-4514-a64d-b323b0a48145\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") "
Nov 03 13:15:19.911726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911616    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca\") pod \"bfccd7ee-752b-4989-9100-58fb95133460\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") "
Nov 03 13:15:19.911726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911638    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b\" (UniqueName: \"kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b\") pod \"b3d3e58a-3062-4514-a64d-b323b0a48145\" (UID: \"b3d3e58a-3062-4514-a64d-b323b0a48145\") "
Nov 03 13:15:19.911726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911689    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf\" (UniqueName: \"kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf\") pod \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") "
Nov 03 13:15:19.911726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911710    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca\") pod \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") "
Nov 03 13:15:19.911989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911784    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj\" (UniqueName: \"kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj\") pod \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\" (UID: \"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330\") "
Nov 03 13:15:19.911989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911865    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq\") pod \"bfccd7ee-752b-4989-9100-58fb95133460\" (UID: \"bfccd7ee-752b-4989-9100-58fb95133460\") "
Nov 03 13:15:19.911989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911919    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca\") pod \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\" (UID: \"a313ff7a-0080-4b91-b3c9-074f7ab14a47\") "
Nov 03 13:15:19.911989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911974    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca" pod "bfccd7ee-752b-4989-9100-58fb95133460" (UID: "bfccd7ee-752b-4989-9100-58fb95133460"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:19.912190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.911980    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca" pod "b3d3e58a-3062-4514-a64d-b323b0a48145" (UID: "b3d3e58a-3062-4514-a64d-b323b0a48145"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:19.912484 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.912459    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca" pod "a313ff7a-0080-4b91-b3c9-074f7ab14a47" (UID: "a313ff7a-0080-4b91-b3c9-074f7ab14a47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:19.912581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.912480    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d3e58a-3062-4514-a64d-b323b0a48145-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.912581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.912484    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca" pod "33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" (UID: "33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:19.912581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.912502    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a434-5281-4346-bea0-09b1fd978ea1-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.912581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.912516    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfccd7ee-752b-4989-9100-58fb95133460-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.912581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.912547    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntmt\" (UniqueName: \"kubernetes.io/projected/0d22a434-5281-4346-bea0-09b1fd978ea1-kube-api-access-5ntmt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:19.914034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.913980    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b" pod "b3d3e58a-3062-4514-a64d-b323b0a48145" (UID: "b3d3e58a-3062-4514-a64d-b323b0a48145"). InnerVolumeSpecName "kube-api-access-js84b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:19.914160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.914035    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj" pod "33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" (UID: "33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330"). InnerVolumeSpecName "kube-api-access-2rvqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:19.914318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.914295    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf" pod "a313ff7a-0080-4b91-b3c9-074f7ab14a47" (UID: "a313ff7a-0080-4b91-b3c9-074f7ab14a47"). InnerVolumeSpecName "kube-api-access-872jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:19.914402 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:19.914386    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq" pod "bfccd7ee-752b-4989-9100-58fb95133460" (UID: "bfccd7ee-752b-4989-9100-58fb95133460"). InnerVolumeSpecName "kube-api-access-2xnlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:20.014019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.013980    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js84b\" (UniqueName: \"kubernetes.io/projected/b3d3e58a-3062-4514-a64d-b323b0a48145-kube-api-access-js84b\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.014019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.014021    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872jf\" (UniqueName: \"kubernetes.io/projected/a313ff7a-0080-4b91-b3c9-074f7ab14a47-kube-api-access-872jf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.014246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.014037    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.014246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.014051    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvqj\" (UniqueName: \"kubernetes.io/projected/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330-kube-api-access-2rvqj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.014246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.014077    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/bfccd7ee-752b-4989-9100-58fb95133460-kube-api-access-2xnlq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.014246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.014090    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a313ff7a-0080-4b91-b3c9-074f7ab14a47-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:20.405555 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.405520    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"]
Nov 03 13:15:20.446505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.446474    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895195-szhfg"]
Nov 03 13:15:20.576475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.576445    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" path="/var/lib/kubelet/pods/6d187689-6a91-4121-a41a-6e7cc5368be2/volumes"
Nov 03 13:15:20.576989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.576970    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" path="/var/lib/kubelet/pods/e5017aac-1e27-4654-af4f-912ec2f334d0/volumes"
Nov 03 13:15:20.577407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.577388    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" path="/var/lib/kubelet/pods/e61a57bb-d311-423f-85ba-aead972ea4fb/volumes"
Nov 03 13:15:20.592548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.592521    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv" event={"ID":"33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330","Type":"ContainerDied","Data":"61a47fe458d0e1f4f2e35c236ecf556d2d0decab300d374b283166b6bc5f2fcd"}
Nov 03 13:15:20.592687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.592579    4546 scope.go:122] "RemoveContainer" containerID="2244ee257cc2a26fb93a4973315f40d215c79ae7796d6f586c0c42fdcf0976f9"
Nov 03 13:15:20.592687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.592537    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"
Nov 03 13:15:20.597264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.597240    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"
Nov 03 13:15:20.597382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.597272    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt" event={"ID":"0d22a434-5281-4346-bea0-09b1fd978ea1","Type":"ContainerDied","Data":"47b74220f90cea35a89bfc6f7c7b688c076903147f835f777d62f768b11ce55a"}
Nov 03 13:15:20.598918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.598900    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"
Nov 03 13:15:20.599015 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.598929    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd" event={"ID":"b3d3e58a-3062-4514-a64d-b323b0a48145","Type":"ContainerDied","Data":"7a1a674388c0fb9377b52fdd18f8935d7d436cf99f21196b118e5d99731d8daf"}
Nov 03 13:15:20.605842 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.605817    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj" event={"ID":"bfccd7ee-752b-4989-9100-58fb95133460","Type":"ContainerDied","Data":"1f449b2267618949dbc00de25827afaf3e003b99187746c0afa162bbdfa1c488"}
Nov 03 13:15:20.605960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.605898    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"
Nov 03 13:15:20.606311 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.606296    4546 scope.go:122] "RemoveContainer" containerID="60c056e97a92aaa40c26767fc95ea9166fda3b4078da89a3d0e2cfb7cbc14ad7"
Nov 03 13:15:20.611816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.607206    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f" event={"ID":"a313ff7a-0080-4b91-b3c9-074f7ab14a47","Type":"ContainerDied","Data":"98ca836ccda45f1c00e1f1191efe9f822ce2979cfe3918d128acbdaff0631dca"}
Nov 03 13:15:20.611816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.607261    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"
Nov 03 13:15:20.616813 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.616795    4546 scope.go:122] "RemoveContainer" containerID="57146334492f18a408475708aa9ae74319f806868ff2e888e2b9ea597f6194a3"
Nov 03 13:15:20.625801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.625782    4546 scope.go:122] "RemoveContainer" containerID="4a1ffefb4d091a5b7ae3a14dddf173fbf9565eedad308c7557940a5efec12658"
Nov 03 13:15:20.634315 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.634286    4546 scope.go:122] "RemoveContainer" containerID="3e7c55952382983e9af56f33f3dfe6dfadb6e486e305e0fa47b71c870e18612c"
Nov 03 13:15:20.707198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.707106    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"]
Nov 03 13:15:20.886403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.886373    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:20.924649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.924613    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z\" (UniqueName: \"kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z\") pod \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") "
Nov 03 13:15:20.924863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.924794    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca\") pod \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\" (UID: \"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6\") "
Nov 03 13:15:20.925251 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.925220    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca" pod "fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" (UID: "fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:20.927008 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.926983    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z" pod "fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" (UID: "fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6"). InnerVolumeSpecName "kube-api-access-pc57z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:20.969793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:20.969765    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"]
Nov 03 13:15:21.043668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.043633    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc57z\" (UniqueName: \"kubernetes.io/projected/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-kube-api-access-pc57z\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.044401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.044366    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.113423 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.113391    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"]
Nov 03 13:15:21.120062 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.120032    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e0615c-29895195-h8phj"]
Nov 03 13:15:21.143924 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.143899    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:21.235401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.234706    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"]
Nov 03 13:15:21.240164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.240112    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94e247-29895195-hcpfv"]
Nov 03 13:15:21.247121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.247088    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca\") pod \"144e51e1-4793-4c76-a1d5-c5003a0774a7\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") "
Nov 03 13:15:21.247350 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.247333    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq\") pod \"144e51e1-4793-4c76-a1d5-c5003a0774a7\" (UID: \"144e51e1-4793-4c76-a1d5-c5003a0774a7\") "
Nov 03 13:15:21.247533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.247509    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca" pod "144e51e1-4793-4c76-a1d5-c5003a0774a7" (UID: "144e51e1-4793-4c76-a1d5-c5003a0774a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:21.247856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.247831    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e51e1-4793-4c76-a1d5-c5003a0774a7-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.249609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.249586    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq" pod "144e51e1-4793-4c76-a1d5-c5003a0774a7" (UID: "144e51e1-4793-4c76-a1d5-c5003a0774a7"). InnerVolumeSpecName "kube-api-access-xnlwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:21.334377 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.334344    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"]
Nov 03 13:15:21.344285 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.344252    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-q2j5f"]
Nov 03 13:15:21.349120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.349087    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/144e51e1-4793-4c76-a1d5-c5003a0774a7-kube-api-access-xnlwq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.442703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.442669    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"]
Nov 03 13:15:21.448310 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.448282    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ab27b8-29895195-6vchd"]
Nov 03 13:15:21.553051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.552970    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"]
Nov 03 13:15:21.558856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.558825    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-02f899-29895195-bzjbt"]
Nov 03 13:15:21.618143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.618115    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"
Nov 03 13:15:21.618328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.618176    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm" event={"ID":"144e51e1-4793-4c76-a1d5-c5003a0774a7","Type":"ContainerDied","Data":"ad7d9d6e949757d5938dd3b721830707b1a5367a36bf840787213de02ae66961"}
Nov 03 13:15:21.618328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.618230    4546 scope.go:122] "RemoveContainer" containerID="07146d9c4110aa67b28fc06d99c09cffa5bb05b26dd59ff438f2fc105d5ff2d0"
Nov 03 13:15:21.623547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.623525    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"
Nov 03 13:15:21.623676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.623514    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk" event={"ID":"fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6","Type":"ContainerDied","Data":"ac58062e249d4048a1ac029d420269176a20bec396ae76759d3d2a1dd4debe53"}
Nov 03 13:15:21.629793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.629770    4546 scope.go:122] "RemoveContainer" containerID="3cfc20ae39c6bc09c9c2981226ae28fc317d26b016f45c62d9076a13a1f25cbc"
Nov 03 13:15:21.774774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.774664    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"]
Nov 03 13:15:21.778087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.778058    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bb1b51-29895195-gxbwk"]
Nov 03 13:15:21.877717 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.877619    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"]
Nov 03 13:15:21.881392 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:21.881367    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a4489c-29895195-j5kkm"]
Nov 03 13:15:22.573273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.573243    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" path="/var/lib/kubelet/pods/0d22a434-5281-4346-bea0-09b1fd978ea1/volumes"
Nov 03 13:15:22.573722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.573708    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" path="/var/lib/kubelet/pods/144e51e1-4793-4c76-a1d5-c5003a0774a7/volumes"
Nov 03 13:15:22.574202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.574189    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" path="/var/lib/kubelet/pods/33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330/volumes"
Nov 03 13:15:22.574929 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.574913    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" path="/var/lib/kubelet/pods/a313ff7a-0080-4b91-b3c9-074f7ab14a47/volumes"
Nov 03 13:15:22.575320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.575307    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" path="/var/lib/kubelet/pods/b3d3e58a-3062-4514-a64d-b323b0a48145/volumes"
Nov 03 13:15:22.575678 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.575665    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfccd7ee-752b-4989-9100-58fb95133460" path="/var/lib/kubelet/pods/bfccd7ee-752b-4989-9100-58fb95133460/volumes"
Nov 03 13:15:22.576306 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:22.576291    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" path="/var/lib/kubelet/pods/fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6/volumes"
Nov 03 13:15:26.281006 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:26.280973    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"]
Nov 03 13:15:26.281399 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:26.281232    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" containerID="cri-o://04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4" gracePeriod=30
Nov 03 13:15:26.948470 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:26.948425    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d" podUID="6c9283de-81a2-4464-beef-704698c0148f" containerName="main" containerID="cri-o://9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d" gracePeriod=45
Nov 03 13:15:27.223843 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.223817    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:15:27.316625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.316599    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47\" (UniqueName: \"kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47\") pod \"6c9283de-81a2-4464-beef-704698c0148f\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") "
Nov 03 13:15:27.317037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.316653    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm\") pod \"6c9283de-81a2-4464-beef-704698c0148f\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") "
Nov 03 13:15:27.317037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.316702    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir\") pod \"6c9283de-81a2-4464-beef-704698c0148f\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") "
Nov 03 13:15:27.317037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.316771    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs\") pod \"6c9283de-81a2-4464-beef-704698c0148f\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") "
Nov 03 13:15:27.317037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.316867    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home\") pod \"6c9283de-81a2-4464-beef-704698c0148f\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") "
Nov 03 13:15:27.317037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317017    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir" pod "6c9283de-81a2-4464-beef-704698c0148f" (UID: "6c9283de-81a2-4464-beef-704698c0148f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:27.317037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317034    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache\") pod \"6c9283de-81a2-4464-beef-704698c0148f\" (UID: \"6c9283de-81a2-4464-beef-704698c0148f\") "
Nov 03 13:15:27.317373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317147    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home" pod "6c9283de-81a2-4464-beef-704698c0148f" (UID: "6c9283de-81a2-4464-beef-704698c0148f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:27.317373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317241    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache" pod "6c9283de-81a2-4464-beef-704698c0148f" (UID: "6c9283de-81a2-4464-beef-704698c0148f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:27.317686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317666    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:27.317766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317718    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:27.317766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.317735    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:27.318926 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.318903    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs" pod "6c9283de-81a2-4464-beef-704698c0148f" (UID: "6c9283de-81a2-4464-beef-704698c0148f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:27.319060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.319042    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm" pod "6c9283de-81a2-4464-beef-704698c0148f" (UID: "6c9283de-81a2-4464-beef-704698c0148f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:27.319119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.319055    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47" pod "6c9283de-81a2-4464-beef-704698c0148f" (UID: "6c9283de-81a2-4464-beef-704698c0148f"). InnerVolumeSpecName "kube-api-access-gmm47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:27.418846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.418817    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmm47\" (UniqueName: \"kubernetes.io/projected/6c9283de-81a2-4464-beef-704698c0148f-kube-api-access-gmm47\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:27.418846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.418841    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9283de-81a2-4464-beef-704698c0148f-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:27.418846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.418849    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9283de-81a2-4464-beef-704698c0148f-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:27.677126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.677091    4546 generic.go:356] "Generic (PLEG): container finished" podID="6c9283de-81a2-4464-beef-704698c0148f" containerID="9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d" exitCode=0
Nov 03 13:15:27.677308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.677129    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"
Nov 03 13:15:27.677308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.677179    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d" event={"ID":"6c9283de-81a2-4464-beef-704698c0148f","Type":"ContainerDied","Data":"9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d"}
Nov 03 13:15:27.677308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.677223    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d" event={"ID":"6c9283de-81a2-4464-beef-704698c0148f","Type":"ContainerDied","Data":"e8bf5da966e6bec97df3f0d399e53f3ac2e6bfccd170146b656c53c0225740ab"}
Nov 03 13:15:27.677308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.677244    4546 scope.go:122] "RemoveContainer" containerID="9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d"
Nov 03 13:15:27.688542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.688523    4546 scope.go:122] "RemoveContainer" containerID="9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d"
Nov 03 13:15:27.688813 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:27.688791    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d\": container with ID starting with 9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d not found: ID does not exist" containerID="9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d"
Nov 03 13:15:27.688878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.688823    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d"} err="failed to get container status \"9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d\": rpc error: code = NotFound desc = could not find container \"9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d\": container with ID starting with 9d25de920a7661f5672297ab5aa8e9333adc35f5aa5c363622cda4e7cdb9bd7d not found: ID does not exist"
Nov 03 13:15:27.831551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.831526    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"]
Nov 03 13:15:27.877500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:27.877477    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8570a0/rate-test-model-8570a0-kserve-78d6dfffdf-p4q7d"]
Nov 03 13:15:28.382847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.382817    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"]
Nov 03 13:15:28.383343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383330    4546 cpu_manager.go:414] "containerMap: removing container" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy"
Nov 03 13:15:28.383382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383344    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy"
Nov 03 13:15:28.383382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383355    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:28.383382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383361    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:28.383382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383370    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.383382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383375    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383385    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383390    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383397    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383401    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383411    4546 cpu_manager.go:414] "containerMap: removing container" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383415    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383420    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383425    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383431    4546 cpu_manager.go:414] "containerMap: removing container" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383436    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383441    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383445    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383454    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6c9283de-81a2-4464-beef-704698c0148f" containerName="main"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383459    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9283de-81a2-4464-beef-704698c0148f" containerName="main"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383467    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383473    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383479    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383483    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383494    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383498    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383505    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383509    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383517    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383521    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383527    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383531    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383536    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383540    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383547    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383551    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383559    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.383550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383564    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383572    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383577    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383585    4546 cpu_manager.go:414] "containerMap: removing container" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383589    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383597    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383601    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383606    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383610    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383616    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383620    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383630    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383634    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383705    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383713    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383721    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383726    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383742    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383767    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383773    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383779    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383787    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="213e49b5-465a-480e-955e-c6b83f42194b" containerName="istio-proxy"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383793    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383798    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383803    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383810    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383815    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383821    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383825    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383830    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383834    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383839    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9283de-81a2-4464-beef-704698c0148f" containerName="main"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383844    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383848    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383853    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383860    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383864    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383869    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:28.384557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.383875    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:28.390875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.390849    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.429607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.429582    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.429736 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.429721    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.429877 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.429857    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7p8n\" (UniqueName: \"kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.441039 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.440691    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:15:28.442700 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.442666    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"]
Nov 03 13:15:28.484184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.484160    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:15:28.484184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.484159    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:15:28.484184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.484179    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:15:28.531025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.530998    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7p8n\" (UniqueName: \"kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.531170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.531049    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.531244 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.531185    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.531722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.531684    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.533600 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.533571    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.576035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.576012    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9283de-81a2-4464-beef-704698c0148f" path="/var/lib/kubelet/pods/6c9283de-81a2-4464-beef-704698c0148f/volumes"
Nov 03 13:15:28.584196 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.584166    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7p8n\" (UniqueName: \"kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n\") pod \"maas-api-revoke-sub-921daae4-qzffn\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:28.705684 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:28.705602    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:29.073601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:29.073570    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"]
Nov 03 13:15:29.074272 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:29.074247    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51747e1_6638_441f_9db0_04e57347cf7c.slice/crio-481f95bbeb6ae76c93e5eaf28711fcf34aa00a5c3404f736b9a97f29c22bbe75 WatchSource:0}: Error finding container 481f95bbeb6ae76c93e5eaf28711fcf34aa00a5c3404f736b9a97f29c22bbe75: Status 404 returned error can't find the container with id 481f95bbeb6ae76c93e5eaf28711fcf34aa00a5c3404f736b9a97f29c22bbe75
Nov 03 13:15:29.695003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:29.694922    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn" event={"ID":"f51747e1-6638-441f-9db0-04e57347cf7c","Type":"ContainerStarted","Data":"dcb0853a89b86e5a245bc65e9637578ff023bb7e20c823864d4d54dc32d67db2"}
Nov 03 13:15:29.695003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:29.694969    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn" event={"ID":"f51747e1-6638-441f-9db0-04e57347cf7c","Type":"ContainerStarted","Data":"481f95bbeb6ae76c93e5eaf28711fcf34aa00a5c3404f736b9a97f29c22bbe75"}
Nov 03 13:15:29.801774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:29.801676    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn" podStartSLOduration=1.8016598560000001 podStartE2EDuration="1.801659856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:29.800020488 +0000 UTC m=+5071.371604417" watchObservedRunningTime="2026-11-03 13:15:29.801659856 +0000 UTC m=+5071.373243770"
Nov 03 13:15:30.281771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:30.281724    4546 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:15:30.281955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:30.281806    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:15:30.703790 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:30.703679    4546 generic.go:356] "Generic (PLEG): container finished" podID="f51747e1-6638-441f-9db0-04e57347cf7c" containerID="dcb0853a89b86e5a245bc65e9637578ff023bb7e20c823864d4d54dc32d67db2" exitCode=0
Nov 03 13:15:30.703790 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:30.703743    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn" event={"ID":"f51747e1-6638-441f-9db0-04e57347cf7c","Type":"ContainerDied","Data":"dcb0853a89b86e5a245bc65e9637578ff023bb7e20c823864d4d54dc32d67db2"}
Nov 03 13:15:31.548872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.548797    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 13:15:31.668073 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668042    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668106    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668130    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668169    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668198    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668257    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668295    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.668557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668303    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668441    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668483    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.668557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668506    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket\") pod \"69594464-d692-4da3-8dd3-d31e710455c8\" (UID: \"69594464-d692-4da3-8dd3-d31e710455c8\") "
Nov 03 13:15:31.668873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668569    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.668873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668715    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.669172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.668946    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:31.669407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.669380    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69594464-d692-4da3-8dd3-d31e710455c8-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.669407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.669408    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.669615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.669422    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.669615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.669436    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.669615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.669449    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.670803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.670781    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:31.670803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.670786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "kube-api-access-qgbjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:31.670921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.670811    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:15:31.670921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.670853    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token" pod "69594464-d692-4da3-8dd3-d31e710455c8" (UID: "69594464-d692-4da3-8dd3-d31e710455c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:31.713260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.713227    4546 generic.go:356] "Generic (PLEG): container finished" podID="69594464-d692-4da3-8dd3-d31e710455c8" containerID="04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4" exitCode=0
Nov 03 13:15:31.713699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.713287    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"
Nov 03 13:15:31.713699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.713312    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" event={"ID":"69594464-d692-4da3-8dd3-d31e710455c8","Type":"ContainerDied","Data":"04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4"}
Nov 03 13:15:31.713699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.713380    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r" event={"ID":"69594464-d692-4da3-8dd3-d31e710455c8","Type":"ContainerDied","Data":"21deedc0b92283e0d24ab7c1a69edd8d7d0e527f6ce904e0f09b55e7638f4cf4"}
Nov 03 13:15:31.713699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.713415    4546 scope.go:122] "RemoveContainer" containerID="04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4"
Nov 03 13:15:31.729534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.729514    4546 scope.go:122] "RemoveContainer" containerID="ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"
Nov 03 13:15:31.741349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.741327    4546 scope.go:122] "RemoveContainer" containerID="04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4"
Nov 03 13:15:31.741721 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:31.741691    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4\": container with ID starting with 04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4 not found: ID does not exist" containerID="04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4"
Nov 03 13:15:31.741821 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.741737    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4"} err="failed to get container status \"04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4\": rpc error: code = NotFound desc = could not find container \"04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4\": container with ID starting with 04d3f81ed5735fba5917d4a468029666b0be5572dcbf2eb11b85c450850563e4 not found: ID does not exist"
Nov 03 13:15:31.741821 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.741782    4546 scope.go:122] "RemoveContainer" containerID="ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"
Nov 03 13:15:31.742093 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:31.742071    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108\": container with ID starting with ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108 not found: ID does not exist" containerID="ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"
Nov 03 13:15:31.742149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.742101    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108"} err="failed to get container status \"ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108\": rpc error: code = NotFound desc = could not find container \"ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108\": container with ID starting with ceffe36be851f69f053a8d446b3d769078f5713bb10f23fba906705ccb031108 not found: ID does not exist"
Nov 03 13:15:31.770629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.770594    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbjn\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-kube-api-access-qgbjn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.770823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.770632    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69594464-d692-4da3-8dd3-d31e710455c8-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.770823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.770651    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69594464-d692-4da3-8dd3-d31e710455c8-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.770823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.770664    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69594464-d692-4da3-8dd3-d31e710455c8-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.862036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.862009    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:31.868458 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.868434    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"]
Nov 03 13:15:31.874302 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.874278    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1250b3-openshift-default-76f96b6655-nfs5r"]
Nov 03 13:15:31.973013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.972982    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca\") pod \"f51747e1-6638-441f-9db0-04e57347cf7c\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") "
Nov 03 13:15:31.973194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.973040    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n\" (UniqueName: \"kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n\") pod \"f51747e1-6638-441f-9db0-04e57347cf7c\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") "
Nov 03 13:15:31.973235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.973215    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca\") pod \"f51747e1-6638-441f-9db0-04e57347cf7c\" (UID: \"f51747e1-6638-441f-9db0-04e57347cf7c\") "
Nov 03 13:15:31.973604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.973536    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca" pod "f51747e1-6638-441f-9db0-04e57347cf7c" (UID: "f51747e1-6638-441f-9db0-04e57347cf7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:31.973863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.973674    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:31.975138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.975115    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca" pod "f51747e1-6638-441f-9db0-04e57347cf7c" (UID: "f51747e1-6638-441f-9db0-04e57347cf7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:31.975277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:31.975253    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n" pod "f51747e1-6638-441f-9db0-04e57347cf7c" (UID: "f51747e1-6638-441f-9db0-04e57347cf7c"). InnerVolumeSpecName "kube-api-access-c7p8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:32.074429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:32.074401    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51747e1-6638-441f-9db0-04e57347cf7c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:32.074429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:32.074427    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7p8n\" (UniqueName: \"kubernetes.io/projected/f51747e1-6638-441f-9db0-04e57347cf7c-kube-api-access-c7p8n\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:32.573392 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:32.573311    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69594464-d692-4da3-8dd3-d31e710455c8" path="/var/lib/kubelet/pods/69594464-d692-4da3-8dd3-d31e710455c8/volumes"
Nov 03 13:15:32.722313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:32.722283    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn" event={"ID":"f51747e1-6638-441f-9db0-04e57347cf7c","Type":"ContainerDied","Data":"481f95bbeb6ae76c93e5eaf28711fcf34aa00a5c3404f736b9a97f29c22bbe75"}
Nov 03 13:15:32.722313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:32.722313    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"
Nov 03 13:15:32.722739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:32.722328    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="481f95bbeb6ae76c93e5eaf28711fcf34aa00a5c3404f736b9a97f29c22bbe75"
Nov 03 13:15:34.400139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:34.400097    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"]
Nov 03 13:15:34.509525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:34.509498    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"]
Nov 03 13:15:37.656444 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:37.656366    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:15:39.246213 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246180    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"]
Nov 03 13:15:39.246701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246687    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f51747e1-6638-441f-9db0-04e57347cf7c" containerName="revoke-keys"
Nov 03 13:15:39.246772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246702    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51747e1-6638-441f-9db0-04e57347cf7c" containerName="revoke-keys"
Nov 03 13:15:39.246772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246719    4546 cpu_manager.go:414] "containerMap: removing container" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy"
Nov 03 13:15:39.246772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246724    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy"
Nov 03 13:15:39.246772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246733    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:39.246772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246737    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5017aac-1e27-4654-af4f-912ec2f334d0" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246774    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246782    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246800    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246804    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246824    4546 cpu_manager.go:414] "containerMap: removing container" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246829    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246837    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246841    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246848    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246852    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfccd7ee-752b-4989-9100-58fb95133460" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246863    4546 cpu_manager.go:414] "containerMap: removing container" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246867    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144e51e1-4793-4c76-a1d5-c5003a0774a7" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246872    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246876    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:39.246957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246954    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246965    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="33fe4ebc-7d3b-4ce1-9fc6-d9c8a4e09330" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246973    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d187689-6a91-4121-a41a-6e7cc5368be2" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246980    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61a57bb-d311-423f-85ba-aead972ea4fb" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246986    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a313ff7a-0080-4b91-b3c9-074f7ab14a47" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.246992    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0a31c5-2493-428c-a4e1-2a50fe3eb8e6" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.247000    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d22a434-5281-4346-bea0-09b1fd978ea1" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.247006    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d3e58a-3062-4514-a64d-b323b0a48145" containerName="cleanup"
Nov 03 13:15:39.247404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.247010    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51747e1-6638-441f-9db0-04e57347cf7c" containerName="revoke-keys"
Nov 03 13:15:39.251713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.251693    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.349647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.349617    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:15:39.349647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.349617    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:15:39.349647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.349617    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:15:39.349916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.349809    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:15:39.353004 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.352982    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"]
Nov 03 13:15:39.362925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.362900    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.363047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.363027    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.363158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.363141    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsslc\" (UniqueName: \"kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.400891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.400856    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerName="payload-processing" containerID="cri-o://f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772" gracePeriod=25
Nov 03 13:15:39.464415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.464386    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.464558 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.464458    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.464558 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.464509    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsslc\" (UniqueName: \"kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.465233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.465157    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.467117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.467096    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.510739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.510670    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerName="payload-pre-processing" containerID="cri-o://818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c" gracePeriod=25
Nov 03 13:15:39.518047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.518024    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsslc\" (UniqueName: \"kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc\") pod \"maas-api-revoke-sub-553d398e-k2hd8\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.564873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.564842    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:39.934332 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:39.934306    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"]
Nov 03 13:15:39.935473 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:39.935441    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f23854_8d8e_466b_af64_a1b69bd2aa4e.slice/crio-2ec45e0c59fd13d519bd399a41f1295c8b3af0ae2f78c43b7414a349189e26f5 WatchSource:0}: Error finding container 2ec45e0c59fd13d519bd399a41f1295c8b3af0ae2f78c43b7414a349189e26f5: Status 404 returned error can't find the container with id 2ec45e0c59fd13d519bd399a41f1295c8b3af0ae2f78c43b7414a349189e26f5
Nov 03 13:15:40.775515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:40.775482    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8" event={"ID":"71f23854-8d8e-466b-af64-a1b69bd2aa4e","Type":"ContainerStarted","Data":"a126f00a4c1aa0a6c22791ace84206057048fbab9694e4481664dda54c5a1d06"}
Nov 03 13:15:40.775951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:40.775544    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8" event={"ID":"71f23854-8d8e-466b-af64-a1b69bd2aa4e","Type":"ContainerStarted","Data":"2ec45e0c59fd13d519bd399a41f1295c8b3af0ae2f78c43b7414a349189e26f5"}
Nov 03 13:15:40.890829 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:40.890768    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8" podStartSLOduration=1.890735573 podStartE2EDuration="1.890735573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:40.886601965 +0000 UTC m=+5082.458185878" watchObservedRunningTime="2026-11-03 13:15:40.890735573 +0000 UTC m=+5082.462319489"
Nov 03 13:15:41.782844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:41.782809    4546 generic.go:356] "Generic (PLEG): container finished" podID="71f23854-8d8e-466b-af64-a1b69bd2aa4e" containerID="a126f00a4c1aa0a6c22791ace84206057048fbab9694e4481664dda54c5a1d06" exitCode=0
Nov 03 13:15:41.783341 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:41.782889    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8" event={"ID":"71f23854-8d8e-466b-af64-a1b69bd2aa4e","Type":"ContainerDied","Data":"a126f00a4c1aa0a6c22791ace84206057048fbab9694e4481664dda54c5a1d06"}
Nov 03 13:15:42.934505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:42.934483    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:43.105585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.105516    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca\") pod \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") "
Nov 03 13:15:43.105585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.105583    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc\" (UniqueName: \"kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc\") pod \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") "
Nov 03 13:15:43.105819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.105658    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca\") pod \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\" (UID: \"71f23854-8d8e-466b-af64-a1b69bd2aa4e\") "
Nov 03 13:15:43.105971 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.105943    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca" pod "71f23854-8d8e-466b-af64-a1b69bd2aa4e" (UID: "71f23854-8d8e-466b-af64-a1b69bd2aa4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:43.107848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.107822    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca" pod "71f23854-8d8e-466b-af64-a1b69bd2aa4e" (UID: "71f23854-8d8e-466b-af64-a1b69bd2aa4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:43.108043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.108022    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc" pod "71f23854-8d8e-466b-af64-a1b69bd2aa4e" (UID: "71f23854-8d8e-466b-af64-a1b69bd2aa4e"). InnerVolumeSpecName "kube-api-access-wsslc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:43.206948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.206921    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:43.206948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.206948    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsslc\" (UniqueName: \"kubernetes.io/projected/71f23854-8d8e-466b-af64-a1b69bd2aa4e-kube-api-access-wsslc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:43.207103 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.206962    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f23854-8d8e-466b-af64-a1b69bd2aa4e-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:43.796793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.796739    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"
Nov 03 13:15:43.796793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.796778    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8" event={"ID":"71f23854-8d8e-466b-af64-a1b69bd2aa4e","Type":"ContainerDied","Data":"2ec45e0c59fd13d519bd399a41f1295c8b3af0ae2f78c43b7414a349189e26f5"}
Nov 03 13:15:43.797049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:43.796819    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec45e0c59fd13d519bd399a41f1295c8b3af0ae2f78c43b7414a349189e26f5"
Nov 03 13:15:44.068184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:44.068103    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"]
Nov 03 13:15:44.075957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:44.075934    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80092f4b-n9j8m"]
Nov 03 13:15:44.388663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:44.388576    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:44.572595 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:44.572564    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa7f2af-4899-4d3b-8159-81a7b19621e3" path="/var/lib/kubelet/pods/aaa7f2af-4899-4d3b-8159-81a7b19621e3/volumes"
Nov 03 13:15:45.395190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:45.395152    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:49.743314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:49.743272    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"]
Nov 03 13:15:49.743898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:49.743539    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy" containerID="cri-o://de675847c2663111f6e86a4df3f033bd0d53a2c5e94bb68d9f275113ea0faaa5" gracePeriod=30
Nov 03 13:15:50.851670 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.851638    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"]
Nov 03 13:15:50.852228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.852214    4546 cpu_manager.go:414] "containerMap: removing container" podUID="71f23854-8d8e-466b-af64-a1b69bd2aa4e" containerName="revoke-keys"
Nov 03 13:15:50.852273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.852229    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f23854-8d8e-466b-af64-a1b69bd2aa4e" containerName="revoke-keys"
Nov 03 13:15:50.852273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.852244    4546 cpu_manager.go:414] "containerMap: removing container" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy"
Nov 03 13:15:50.852273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.852249    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy"
Nov 03 13:15:50.852376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.852349    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f23854-8d8e-466b-af64-a1b69bd2aa4e" containerName="revoke-keys"
Nov 03 13:15:50.852376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.852359    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="69594464-d692-4da3-8dd3-d31e710455c8" containerName="istio-proxy"
Nov 03 13:15:50.858918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.858879    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:50.897519 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.897487    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:50.897737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.897714    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:50.897811 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.897794    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnns\" (UniqueName: \"kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:50.953024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.952991    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:15:50.956387 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.956360    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"]
Nov 03 13:15:50.990597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.990565    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:15:50.990781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.990627    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:15:50.990781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.990631    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:15:50.999327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.999306    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:50.999379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.999349    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnns\" (UniqueName: \"kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:50.999512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:50.999488    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:51.000081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.000062    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:51.001960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.001942    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:51.053660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.053631    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnns\" (UniqueName: \"kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns\") pod \"maas-api-revoke-sub-60f25ca0-62v7p\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:51.178082 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.177989    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:51.550471 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.550438    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"]
Nov 03 13:15:51.551803 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:15:51.551774    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e1b2d2_3a29_4294_9cdb_e6bf27154d1d.slice/crio-12c960201510e91592d1d021360f701befd71f403f534c9124fddd1b39f675b7 WatchSource:0}: Error finding container 12c960201510e91592d1d021360f701befd71f403f534c9124fddd1b39f675b7: Status 404 returned error can't find the container with id 12c960201510e91592d1d021360f701befd71f403f534c9124fddd1b39f675b7
Nov 03 13:15:51.851076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.851046    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p" event={"ID":"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d","Type":"ContainerStarted","Data":"921d0cd3f956c38065e3d97f1e09c75befe11c503e8be69095fe413b79bfa494"}
Nov 03 13:15:51.851209 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.851101    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p" event={"ID":"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d","Type":"ContainerStarted","Data":"12c960201510e91592d1d021360f701befd71f403f534c9124fddd1b39f675b7"}
Nov 03 13:15:51.963006 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:51.962946    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p" podStartSLOduration=1.962931813 podStartE2EDuration="1.962931813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:51.960579217 +0000 UTC m=+5093.532163142" watchObservedRunningTime="2026-11-03 13:15:51.962931813 +0000 UTC m=+5093.534515723"
Nov 03 13:15:52.859143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:52.859109    4546 generic.go:356] "Generic (PLEG): container finished" podID="03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" containerID="921d0cd3f956c38065e3d97f1e09c75befe11c503e8be69095fe413b79bfa494" exitCode=0
Nov 03 13:15:52.859327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:52.859192    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p" event={"ID":"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d","Type":"ContainerDied","Data":"921d0cd3f956c38065e3d97f1e09c75befe11c503e8be69095fe413b79bfa494"}
Nov 03 13:15:54.023217 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.023186    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:54.045779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.045735    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"]
Nov 03 13:15:54.091788 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.091765    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2dfddc-vj5nw"]
Nov 03 13:15:54.134968 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.134909    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns\" (UniqueName: \"kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns\") pod \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") "
Nov 03 13:15:54.135083 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.134967    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca\") pod \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") "
Nov 03 13:15:54.135083 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.135003    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca\") pod \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\" (UID: \"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d\") "
Nov 03 13:15:54.135330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.135306    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca" pod "03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" (UID: "03e1b2d2-3a29-4294-9cdb-e6bf27154d1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:54.137196 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.137169    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca" pod "03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" (UID: "03e1b2d2-3a29-4294-9cdb-e6bf27154d1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:54.137288 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.137202    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns" pod "03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" (UID: "03e1b2d2-3a29-4294-9cdb-e6bf27154d1d"). InnerVolumeSpecName "kube-api-access-ptnns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:54.237017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.236987    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnns\" (UniqueName: \"kubernetes.io/projected/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-kube-api-access-ptnns\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:54.237017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.237016    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:54.237219 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.237026    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:54.388579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.388497    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:54.573283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.573250    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ddc376-a4a7-4f09-acde-de7ed80c5751" path="/var/lib/kubelet/pods/25ddc376-a4a7-4f09-acde-de7ed80c5751/volumes"
Nov 03 13:15:54.874456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.874429    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"
Nov 03 13:15:54.874623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.874425    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p" event={"ID":"03e1b2d2-3a29-4294-9cdb-e6bf27154d1d","Type":"ContainerDied","Data":"12c960201510e91592d1d021360f701befd71f403f534c9124fddd1b39f675b7"}
Nov 03 13:15:54.874623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.874530    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12c960201510e91592d1d021360f701befd71f403f534c9124fddd1b39f675b7"
Nov 03 13:15:54.876995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.876972    4546 generic.go:356] "Generic (PLEG): container finished" podID="689340e2-674e-40de-a499-199fada62980" containerID="de675847c2663111f6e86a4df3f033bd0d53a2c5e94bb68d9f275113ea0faaa5" exitCode=0
Nov 03 13:15:54.877103 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.877049    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" event={"ID":"689340e2-674e-40de-a499-199fada62980","Type":"ContainerDied","Data":"de675847c2663111f6e86a4df3f033bd0d53a2c5e94bb68d9f275113ea0faaa5"}
Nov 03 13:15:54.877148 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.877131    4546 scope.go:122] "RemoveContainer" containerID="cf3826cbd97bc3f166929793ebf5fa0e3fda9d5e1f98f62c77653b0a00b668c1"
Nov 03 13:15:54.996402 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:54.996375    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 13:15:55.044544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044488    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044607    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044699    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044728    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044791    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044821    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044868    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.044945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044947    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.045354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044974    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket\") pod \"689340e2-674e-40de-a499-199fada62980\" (UID: \"689340e2-674e-40de-a499-199fada62980\") "
Nov 03 13:15:55.045354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.044999    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:55.045354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045229    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:55.045354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045156    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:55.045735 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045407    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:55.045735 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045421    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:55.045898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045695    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.045898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045806    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689340e2-674e-40de-a499-199fada62980-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.046017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045903    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.046017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045919    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.046017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.045934    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.047169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.047138    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "kube-api-access-ckg4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:55.047282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.047198    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:55.047353 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.047293    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:15:55.047516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.047500    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy" pod "689340e2-674e-40de-a499-199fada62980" (UID: "689340e2-674e-40de-a499-199fada62980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:55.147499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.147471    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689340e2-674e-40de-a499-199fada62980-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.147499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.147495    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckg4k\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-kube-api-access-ckg4k\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.147499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.147505    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689340e2-674e-40de-a499-199fada62980-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.147703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.147514    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689340e2-674e-40de-a499-199fada62980-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:55.184444 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.184419    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"]
Nov 03 13:15:55.394954 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.394874    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:15:55.885465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.885438    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"
Nov 03 13:15:55.885648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.885473    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q" event={"ID":"689340e2-674e-40de-a499-199fada62980","Type":"ContainerDied","Data":"c8c6ee4f82b8fd354e9c93f836acfe226786a79a93af2916e1d0e6faf69be90a"}
Nov 03 13:15:55.885648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:55.885554    4546 scope.go:122] "RemoveContainer" containerID="de675847c2663111f6e86a4df3f033bd0d53a2c5e94bb68d9f275113ea0faaa5"
Nov 03 13:15:56.040400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:56.040369    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"]
Nov 03 13:15:56.043328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:56.043301    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8570a0-openshift-default-9b7544bdf-5725q"]
Nov 03 13:15:56.572757 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:56.572715    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689340e2-674e-40de-a499-199fada62980" path="/var/lib/kubelet/pods/689340e2-674e-40de-a499-199fada62980/volumes"
Nov 03 13:15:59.673433 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.673367    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 13:15:59.774169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.774144    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 13:15:59.799365 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.799338    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume\") pod \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") "
Nov 03 13:15:59.799529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.799486    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z\" (UniqueName: \"kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z\") pod \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\" (UID: \"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9\") "
Nov 03 13:15:59.799835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.799811    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume" pod "4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" (UID: "4a6d02ad-1e47-42d7-b90c-00e5ad9163d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:59.801724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.801699    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z" pod "4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" (UID: "4a6d02ad-1e47-42d7-b90c-00e5ad9163d9"). InnerVolumeSpecName "kube-api-access-br45z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:59.900188 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.900159    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume\") pod \"ecd012d0-647c-4be4-a45b-0a2504fe9143\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") "
Nov 03 13:15:59.900362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.900230    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56\" (UniqueName: \"kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56\") pod \"ecd012d0-647c-4be4-a45b-0a2504fe9143\" (UID: \"ecd012d0-647c-4be4-a45b-0a2504fe9143\") "
Nov 03 13:15:59.900559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.900531    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume" pod "ecd012d0-647c-4be4-a45b-0a2504fe9143" (UID: "ecd012d0-647c-4be4-a45b-0a2504fe9143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:59.900733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.900714    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br45z\" (UniqueName: \"kubernetes.io/projected/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-kube-api-access-br45z\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:59.900826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.900736    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:59.900826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.900778    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd012d0-647c-4be4-a45b-0a2504fe9143-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:59.902844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.902819    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56" pod "ecd012d0-647c-4be4-a45b-0a2504fe9143" (UID: "ecd012d0-647c-4be4-a45b-0a2504fe9143"). InnerVolumeSpecName "kube-api-access-mzz56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:59.916174 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.916144    4546 generic.go:356] "Generic (PLEG): container finished" podID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerID="818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c" exitCode=0
Nov 03 13:15:59.916318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.916218    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"
Nov 03 13:15:59.916318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.916230    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" event={"ID":"ecd012d0-647c-4be4-a45b-0a2504fe9143","Type":"ContainerDied","Data":"818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c"}
Nov 03 13:15:59.916318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.916314    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj" event={"ID":"ecd012d0-647c-4be4-a45b-0a2504fe9143","Type":"ContainerDied","Data":"e154d36e5a0c69dc5a69a54f52c500d09aa4072ddbf741e41992cb51fd27f2a6"}
Nov 03 13:15:59.916516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.916337    4546 scope.go:122] "RemoveContainer" containerID="818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c"
Nov 03 13:15:59.917943 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.917922    4546 generic.go:356] "Generic (PLEG): container finished" podID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerID="f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772" exitCode=0
Nov 03 13:15:59.918059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.917962    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" event={"ID":"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9","Type":"ContainerDied","Data":"f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772"}
Nov 03 13:15:59.918059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.917996    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf" event={"ID":"4a6d02ad-1e47-42d7-b90c-00e5ad9163d9","Type":"ContainerDied","Data":"b9e0978003ec060e8193b03be7164e7f8e0a62f5521a2fff1eacd581bbbb4116"}
Nov 03 13:15:59.918059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.918057    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"
Nov 03 13:15:59.930091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.930076    4546 scope.go:122] "RemoveContainer" containerID="818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c"
Nov 03 13:15:59.930510 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:59.930492    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c\": container with ID starting with 818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c not found: ID does not exist" containerID="818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c"
Nov 03 13:15:59.930565 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.930517    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c"} err="failed to get container status \"818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c\": rpc error: code = NotFound desc = could not find container \"818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c\": container with ID starting with 818750f4117ba6d36fc1bf22ca62641c279bce3a3b4ce49321f44f070b3fbe0c not found: ID does not exist"
Nov 03 13:15:59.930565 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.930532    4546 scope.go:122] "RemoveContainer" containerID="f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772"
Nov 03 13:15:59.939593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.939562    4546 scope.go:122] "RemoveContainer" containerID="f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772"
Nov 03 13:15:59.939890 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:15:59.939869    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772\": container with ID starting with f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772 not found: ID does not exist" containerID="f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772"
Nov 03 13:15:59.939956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:15:59.939899    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772"} err="failed to get container status \"f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772\": rpc error: code = NotFound desc = could not find container \"f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772\": container with ID starting with f44a5ff8e4a88ba0f87a91cbb61da91a35f883882b3570c57c2d8f014fb2c772 not found: ID does not exist"
Nov 03 13:16:00.002041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.002010    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzz56\" (UniqueName: \"kubernetes.io/projected/ecd012d0-647c-4be4-a45b-0a2504fe9143-kube-api-access-mzz56\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:00.072429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.072394    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"]
Nov 03 13:16:00.077901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.077872    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8570a0-58499d44f5-dslvf"]
Nov 03 13:16:00.181667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.181578    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"]
Nov 03 13:16:00.182645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.182616    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8570a0-85d985db47-7gdhj"]
Nov 03 13:16:00.573978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.573946    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" path="/var/lib/kubelet/pods/4a6d02ad-1e47-42d7-b90c-00e5ad9163d9/volumes"
Nov 03 13:16:00.574337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:00.574324    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" path="/var/lib/kubelet/pods/ecd012d0-647c-4be4-a45b-0a2504fe9143/volumes"
Nov 03 13:16:05.798475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798441    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"]
Nov 03 13:16:05.798948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798936    4546 cpu_manager.go:414] "containerMap: removing container" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy"
Nov 03 13:16:05.798994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798949    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy"
Nov 03 13:16:05.798994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798964    4546 cpu_manager.go:414] "containerMap: removing container" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy"
Nov 03 13:16:05.798994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798969    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy"
Nov 03 13:16:05.798994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798976    4546 cpu_manager.go:414] "containerMap: removing container" podUID="03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" containerName="revoke-keys"
Nov 03 13:16:05.798994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798981    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" containerName="revoke-keys"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.798998    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerName="payload-pre-processing"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799002    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerName="payload-pre-processing"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799011    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerName="payload-processing"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799016    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerName="payload-processing"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799114    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799121    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="689340e2-674e-40de-a499-199fada62980" containerName="istio-proxy"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799126    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" containerName="revoke-keys"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799133    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd012d0-647c-4be4-a45b-0a2504fe9143" containerName="payload-pre-processing"
Nov 03 13:16:05.799169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.799139    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6d02ad-1e47-42d7-b90c-00e5ad9163d9" containerName="payload-processing"
Nov 03 13:16:05.804500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.804482    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:05.856970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.856947    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"]
Nov 03 13:16:05.899152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.899129    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:16:05.899390 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.899372    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:16:05.899472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.899416    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:16:05.899472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.899464    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:16:05.965106 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.965076    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:05.965106 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.965107    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhnq8\" (UniqueName: \"kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:05.965271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:05.965135    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.066563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.066499    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.066563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.066527    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhnq8\" (UniqueName: \"kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.066563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.066561    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.067159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.067136    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.069034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.069011    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.119912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.119887    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhnq8\" (UniqueName: \"kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8\") pod \"maas-api-revoke-sub-a151f42b-zfl2d\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.417904 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.417836    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:06.783176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.783150    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"]
Nov 03 13:16:06.784837 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:16:06.784807    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb33051e_e188_4d16_9b7c_04b3821a972f.slice/crio-0ce76acbfdca1246f202a14bf613cd8eea2ba83954ebc0e100ee825ae5ef81a1 WatchSource:0}: Error finding container 0ce76acbfdca1246f202a14bf613cd8eea2ba83954ebc0e100ee825ae5ef81a1: Status 404 returned error can't find the container with id 0ce76acbfdca1246f202a14bf613cd8eea2ba83954ebc0e100ee825ae5ef81a1
Nov 03 13:16:06.963563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:06.963528    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d" event={"ID":"db33051e-e188-4d16-9b7c-04b3821a972f","Type":"ContainerStarted","Data":"0ce76acbfdca1246f202a14bf613cd8eea2ba83954ebc0e100ee825ae5ef81a1"}
Nov 03 13:16:07.651223 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:07.651179    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:16:07.744110 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:07.744084    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"]
Nov 03 13:16:07.854126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:07.854092    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"]
Nov 03 13:16:07.972122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:07.972080    4546 generic.go:356] "Generic (PLEG): container finished" podID="db33051e-e188-4d16-9b7c-04b3821a972f" containerID="6cf420871d5ceaaa92a2909649de7d4d89f30be468f3d5f2d417960152897c9f" exitCode=0
Nov 03 13:16:07.972518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:07.972159    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d" event={"ID":"db33051e-e188-4d16-9b7c-04b3821a972f","Type":"ContainerDied","Data":"6cf420871d5ceaaa92a2909649de7d4d89f30be468f3d5f2d417960152897c9f"}
Nov 03 13:16:09.116773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.116718    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:09.306213 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.306130    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca\") pod \"db33051e-e188-4d16-9b7c-04b3821a972f\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") "
Nov 03 13:16:09.306383 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.306228    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca\") pod \"db33051e-e188-4d16-9b7c-04b3821a972f\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") "
Nov 03 13:16:09.306383 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.306321    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8\" (UniqueName: \"kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8\") pod \"db33051e-e188-4d16-9b7c-04b3821a972f\" (UID: \"db33051e-e188-4d16-9b7c-04b3821a972f\") "
Nov 03 13:16:09.306580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.306552    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca" pod "db33051e-e188-4d16-9b7c-04b3821a972f" (UID: "db33051e-e188-4d16-9b7c-04b3821a972f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:09.306851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.306836    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:09.308434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.308413    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca" pod "db33051e-e188-4d16-9b7c-04b3821a972f" (UID: "db33051e-e188-4d16-9b7c-04b3821a972f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:09.308531 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.308451    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8" pod "db33051e-e188-4d16-9b7c-04b3821a972f" (UID: "db33051e-e188-4d16-9b7c-04b3821a972f"). InnerVolumeSpecName "kube-api-access-lhnq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:09.408394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.408364    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33051e-e188-4d16-9b7c-04b3821a972f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:09.408394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.408390    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhnq8\" (UniqueName: \"kubernetes.io/projected/db33051e-e188-4d16-9b7c-04b3821a972f-kube-api-access-lhnq8\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:09.985175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.985144    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"
Nov 03 13:16:09.985175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.985155    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d" event={"ID":"db33051e-e188-4d16-9b7c-04b3821a972f","Type":"ContainerDied","Data":"0ce76acbfdca1246f202a14bf613cd8eea2ba83954ebc0e100ee825ae5ef81a1"}
Nov 03 13:16:09.985385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:09.985197    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce76acbfdca1246f202a14bf613cd8eea2ba83954ebc0e100ee825ae5ef81a1"
Nov 03 13:16:10.204847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.204789    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" containerID="cri-o://0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04" gracePeriod=45
Nov 03 13:16:10.470224 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.470202    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 13:16:10.623898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.623210    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs\") pod \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") "
Nov 03 13:16:10.623898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.623280    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj\" (UniqueName: \"kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj\") pod \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") "
Nov 03 13:16:10.623898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.623332    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache\") pod \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") "
Nov 03 13:16:10.623898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.623456    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir\") pod \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") "
Nov 03 13:16:10.623898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.623510    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home\") pod \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") "
Nov 03 13:16:10.623898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.623574    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm\") pod \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\" (UID: \"1a4b8978-01ca-4005-99e5-fc3cbd4edb59\") "
Nov 03 13:16:10.625914 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.625844    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs" pod "1a4b8978-01ca-4005-99e5-fc3cbd4edb59" (UID: "1a4b8978-01ca-4005-99e5-fc3cbd4edb59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:10.626042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.625952    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache" pod "1a4b8978-01ca-4005-99e5-fc3cbd4edb59" (UID: "1a4b8978-01ca-4005-99e5-fc3cbd4edb59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.626374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.626341    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir" pod "1a4b8978-01ca-4005-99e5-fc3cbd4edb59" (UID: "1a4b8978-01ca-4005-99e5-fc3cbd4edb59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.626553 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.626492    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home" pod "1a4b8978-01ca-4005-99e5-fc3cbd4edb59" (UID: "1a4b8978-01ca-4005-99e5-fc3cbd4edb59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.628334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.628308    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm" pod "1a4b8978-01ca-4005-99e5-fc3cbd4edb59" (UID: "1a4b8978-01ca-4005-99e5-fc3cbd4edb59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.629128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.629098    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj" pod "1a4b8978-01ca-4005-99e5-fc3cbd4edb59" (UID: "1a4b8978-01ca-4005-99e5-fc3cbd4edb59"). InnerVolumeSpecName "kube-api-access-xfltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:10.725966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.725931    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.725966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.725959    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfltj\" (UniqueName: \"kubernetes.io/projected/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-kube-api-access-xfltj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.725966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.725969    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.726207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.725977    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.726207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.725986    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.726207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.726012    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4b8978-01ca-4005-99e5-fc3cbd4edb59-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.992338 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.992309    4546 generic.go:356] "Generic (PLEG): container finished" podID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerID="0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04" exitCode=0
Nov 03 13:16:10.992506 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.992355    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"
Nov 03 13:16:10.992506 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.992357    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" event={"ID":"1a4b8978-01ca-4005-99e5-fc3cbd4edb59","Type":"ContainerDied","Data":"0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04"}
Nov 03 13:16:10.992506 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.992485    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" event={"ID":"1a4b8978-01ca-4005-99e5-fc3cbd4edb59","Type":"ContainerDied","Data":"c8ca24a4e136106c727f411d94a8ad5a400990ad1868db58e3be6934b5b7e380"}
Nov 03 13:16:10.992647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:10.992511    4546 scope.go:122] "RemoveContainer" containerID="0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04"
Nov 03 13:16:11.002945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:11.002922    4546 scope.go:122] "RemoveContainer" containerID="0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04"
Nov 03 13:16:11.003222 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:11.003205    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04\": container with ID starting with 0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04 not found: ID does not exist" containerID="0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04"
Nov 03 13:16:11.003269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:11.003230    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04"} err="failed to get container status \"0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04\": rpc error: code = NotFound desc = could not find container \"0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04\": container with ID starting with 0a10c608a4fe680da41651ef0230fe818748c4e4f37d3eb31aae50eec981ba04 not found: ID does not exist"
Nov 03 13:16:11.147398 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:11.147363    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"]
Nov 03 13:16:11.154096 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:11.154070    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft"]
Nov 03 13:16:11.430439 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:11.430342    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/premium-w3-bb1b51-kserve-869dc8dd48-brhft" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 13:16:12.573389 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:12.573359    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" path="/var/lib/kubelet/pods/1a4b8978-01ca-4005-99e5-fc3cbd4edb59/volumes"
Nov 03 13:16:12.744703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:12.744655    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerName="payload-processing" containerID="cri-o://a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5" gracePeriod=25
Nov 03 13:16:12.855439 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:12.855342    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59" podUID="a33009df-f592-4370-b7e0-caea97754eed" containerName="payload-pre-processing" containerID="cri-o://442eece0863d7026e365e08879edcdb2ca4b12c4c2df56cd85c14b1b7913bdc7" gracePeriod=25
Nov 03 13:16:13.007296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.007266    4546 generic.go:356] "Generic (PLEG): container finished" podID="a33009df-f592-4370-b7e0-caea97754eed" containerID="442eece0863d7026e365e08879edcdb2ca4b12c4c2df56cd85c14b1b7913bdc7" exitCode=0
Nov 03 13:16:13.007466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.007342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59" event={"ID":"a33009df-f592-4370-b7e0-caea97754eed","Type":"ContainerDied","Data":"442eece0863d7026e365e08879edcdb2ca4b12c4c2df56cd85c14b1b7913bdc7"}
Nov 03 13:16:13.115274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.115221    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 13:16:13.149315 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.149292    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl\" (UniqueName: \"kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl\") pod \"a33009df-f592-4370-b7e0-caea97754eed\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") "
Nov 03 13:16:13.149481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.149462    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume\") pod \"a33009df-f592-4370-b7e0-caea97754eed\" (UID: \"a33009df-f592-4370-b7e0-caea97754eed\") "
Nov 03 13:16:13.149827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.149800    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume" pod "a33009df-f592-4370-b7e0-caea97754eed" (UID: "a33009df-f592-4370-b7e0-caea97754eed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:13.149941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.149879    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33009df-f592-4370-b7e0-caea97754eed-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:13.151416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.151398    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl" pod "a33009df-f592-4370-b7e0-caea97754eed" (UID: "a33009df-f592-4370-b7e0-caea97754eed"). InnerVolumeSpecName "kube-api-access-4x4cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:13.250866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:13.250836    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x4cl\" (UniqueName: \"kubernetes.io/projected/a33009df-f592-4370-b7e0-caea97754eed-kube-api-access-4x4cl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:14.014130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:14.014099    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"
Nov 03 13:16:14.014130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:14.014117    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59" event={"ID":"a33009df-f592-4370-b7e0-caea97754eed","Type":"ContainerDied","Data":"7f946d378759da4a8556c33e0e2edcf7c83274a346d277dfad406563ef58e130"}
Nov 03 13:16:14.014642 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:14.014179    4546 scope.go:122] "RemoveContainer" containerID="442eece0863d7026e365e08879edcdb2ca4b12c4c2df56cd85c14b1b7913bdc7"
Nov 03 13:16:14.187340 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:14.187309    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"]
Nov 03 13:16:14.191094 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:14.191065    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bb1b51-76c6cddc59-8kl59"]
Nov 03 13:16:14.572229 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:14.572197    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33009df-f592-4370-b7e0-caea97754eed" path="/var/lib/kubelet/pods/a33009df-f592-4370-b7e0-caea97754eed/volumes"
Nov 03 13:16:17.145126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:17.145093    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:21.631621 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:21.631588    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"]
Nov 03 13:16:21.632180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:21.631786    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy" containerID="cri-o://746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0" gracePeriod=30
Nov 03 13:16:26.885848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.885805    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 13:16:26.998818 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.998794    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.998981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.998831    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.998981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.998863    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.998981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.998892    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.998981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.998950    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.999225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999063    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.999225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999122    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.999225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999167    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.999225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999163    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:26.999450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999227    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo\") pod \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\" (UID: \"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d\") "
Nov 03 13:16:26.999450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999243    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:26.999450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999387    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:26.999613 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999496    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:26.999613 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999528    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:26.999723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999629    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:26.999723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999641    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:26.999723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999650    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:26.999723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999658    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:26.999723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:26.999668    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:27.001464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.001438    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:27.001620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.001591    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:27.001739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.001661    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "kube-api-access-z8gr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:27.001856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.001768    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo" pod "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" (UID: "0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:16:27.097072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.097043    4546 generic.go:356] "Generic (PLEG): container finished" podID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerID="746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0" exitCode=0
Nov 03 13:16:27.097264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.097092    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"
Nov 03 13:16:27.097264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.097135    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" event={"ID":"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d","Type":"ContainerDied","Data":"746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0"}
Nov 03 13:16:27.097264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.097204    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4" event={"ID":"0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d","Type":"ContainerDied","Data":"2d68c655ca0b74989711fd870b22657f0eebbeabe70fbeeb4840928e13f00f38"}
Nov 03 13:16:27.097264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.097236    4546 scope.go:122] "RemoveContainer" containerID="746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0"
Nov 03 13:16:27.100523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.100500    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:27.100642 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.100529    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:27.100642 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.100546    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gr5\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-kube-api-access-z8gr5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:27.100642 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.100561    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:27.107953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.107934    4546 scope.go:122] "RemoveContainer" containerID="332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"
Nov 03 13:16:27.116575 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.116559    4546 scope.go:122] "RemoveContainer" containerID="746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0"
Nov 03 13:16:27.116854 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:27.116832    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0\": container with ID starting with 746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0 not found: ID does not exist" containerID="746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0"
Nov 03 13:16:27.116934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.116864    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0"} err="failed to get container status \"746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0\": rpc error: code = NotFound desc = could not find container \"746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0\": container with ID starting with 746ca7431d2965f7111cbc767d2ee83c17c63a0f63eea8bc0b9e67872cfe44c0 not found: ID does not exist"
Nov 03 13:16:27.116934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.116891    4546 scope.go:122] "RemoveContainer" containerID="332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"
Nov 03 13:16:27.117184 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:27.117165    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d\": container with ID starting with 332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d not found: ID does not exist" containerID="332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"
Nov 03 13:16:27.117240 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.117191    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d"} err="failed to get container status \"332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d\": rpc error: code = NotFound desc = could not find container \"332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d\": container with ID starting with 332772a969b1a756379ece86bba79d09bf6fa2a2b556a9094b9145829ca4295d not found: ID does not exist"
Nov 03 13:16:27.144667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.144642    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:27.251184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.251119    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"]
Nov 03 13:16:27.256572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:27.256547    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cacb84-openshift-default-6c69c85dc7-jclb4"]
Nov 03 13:16:28.573580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:28.573539    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" path="/var/lib/kubelet/pods/0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d/volumes"
Nov 03 13:16:29.290096 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:29.290058    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"]
Nov 03 13:16:29.392981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:29.392949    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"]
Nov 03 13:16:32.990230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:32.990199    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 13:16:33.063913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.063858    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume\") pod \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") "
Nov 03 13:16:33.064050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.063933    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz\" (UniqueName: \"kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz\") pod \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\" (UID: \"2e1eef8f-ba1b-4a10-a354-0087daf8d35a\") "
Nov 03 13:16:33.064272 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.064249    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume" pod "2e1eef8f-ba1b-4a10-a354-0087daf8d35a" (UID: "2e1eef8f-ba1b-4a10-a354-0087daf8d35a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:33.064370 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.064356    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:33.066021 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.065998    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz" pod "2e1eef8f-ba1b-4a10-a354-0087daf8d35a" (UID: "2e1eef8f-ba1b-4a10-a354-0087daf8d35a"). InnerVolumeSpecName "kube-api-access-8blrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:33.135587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.135559    4546 generic.go:356] "Generic (PLEG): container finished" podID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerID="a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5" exitCode=0
Nov 03 13:16:33.135713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.135615    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"
Nov 03 13:16:33.135713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.135636    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" event={"ID":"2e1eef8f-ba1b-4a10-a354-0087daf8d35a","Type":"ContainerDied","Data":"a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5"}
Nov 03 13:16:33.135827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.135718    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk" event={"ID":"2e1eef8f-ba1b-4a10-a354-0087daf8d35a","Type":"ContainerDied","Data":"db2b896f4ab9388dcae28c7814c138785a6cd60c6e746a501b1716102a83ced8"}
Nov 03 13:16:33.135827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.135743    4546 scope.go:122] "RemoveContainer" containerID="a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5"
Nov 03 13:16:33.145736 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.145702    4546 scope.go:122] "RemoveContainer" containerID="a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5"
Nov 03 13:16:33.146078 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:33.146055    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5\": container with ID starting with a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5 not found: ID does not exist" containerID="a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5"
Nov 03 13:16:33.146177 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.146082    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5"} err="failed to get container status \"a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5\": rpc error: code = NotFound desc = could not find container \"a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5\": container with ID starting with a9cfab5ae699c46f441a6f11f8d09fec50f631d2aee2fb5d3b93906fde0283c5 not found: ID does not exist"
Nov 03 13:16:33.165899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.165878    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8blrz\" (UniqueName: \"kubernetes.io/projected/2e1eef8f-ba1b-4a10-a354-0087daf8d35a-kube-api-access-8blrz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:33.293522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.293484    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"]
Nov 03 13:16:33.302012 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:33.301983    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bb1b51-58475c868c-n84bk"]
Nov 03 13:16:34.290734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:34.290698    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerName="payload-processing" containerID="cri-o://b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce" gracePeriod=25
Nov 03 13:16:34.393982 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:34.393943    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerName="payload-pre-processing" containerID="cri-o://ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee" gracePeriod=25
Nov 03 13:16:34.572647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:34.572581    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" path="/var/lib/kubelet/pods/2e1eef8f-ba1b-4a10-a354-0087daf8d35a/volumes"
Nov 03 13:16:37.655864 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:37.655831    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:16:38.708028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:38.707990    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:38.708504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:38.707990    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:41.044834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:41.044803    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"]
Nov 03 13:16:41.091690 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:41.091659    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad9cac-8fft4"]
Nov 03 13:16:42.572488 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:42.572444    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf98a2e-e005-458c-9e6a-87718fe4715b" path="/var/lib/kubelet/pods/5bf98a2e-e005-458c-9e6a-87718fe4715b/volumes"
Nov 03 13:16:45.118091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118045    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"]
Nov 03 13:16:45.118572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118558    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118573    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118585    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118590    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118596    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerName="payload-processing"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118601    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerName="payload-processing"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118607    4546 cpu_manager.go:414] "containerMap: removing container" podUID="db33051e-e188-4d16-9b7c-04b3821a972f" containerName="revoke-keys"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118612    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db33051e-e188-4d16-9b7c-04b3821a972f" containerName="revoke-keys"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118623    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy"
Nov 03 13:16:45.118625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118628    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118640    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a33009df-f592-4370-b7e0-caea97754eed" containerName="payload-pre-processing"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118647    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33009df-f592-4370-b7e0-caea97754eed" containerName="payload-pre-processing"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118739    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118761    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4b8978-01ca-4005-99e5-fc3cbd4edb59" containerName="main"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118769    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1eef8f-ba1b-4a10-a354-0087daf8d35a" containerName="payload-processing"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118773    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="db33051e-e188-4d16-9b7c-04b3821a972f" containerName="revoke-keys"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118779    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0956ce2f-6a77-40cc-a5df-f3ce1cfd5a0d" containerName="istio-proxy"
Nov 03 13:16:45.118932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.118784    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33009df-f592-4370-b7e0-caea97754eed" containerName="payload-pre-processing"
Nov 03 13:16:45.121949 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.121932    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.170524 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.170501    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bb1b51-openshift-default-dockercfg-b9g7k\""
Nov 03 13:16:45.220810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.220782    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"]
Nov 03 13:16:45.292744 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.292714    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.292904 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.292775    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.292904 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.292842    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.293017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.292912    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.293017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.292971    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.293090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.293038    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.293090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.293060    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.293194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.293181    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.293235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.293206    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhbm\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394171    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394199    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394281    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394302    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhbm\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394336    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394365    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394383    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394403    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394431    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394820 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394644    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394820 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394681    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394926    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.394994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.394968    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.395399 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.395377    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.396662 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.396642    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.396941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.396914    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.447171 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.447150    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.447463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.447446    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhbm\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:45.735428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:45.735402    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:46.146574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:46.146493    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"]
Nov 03 13:16:46.178563 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:16:46.178535    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fef1e8f_9dfa_4a15_ac7e_cf93e0a65439.slice/crio-e247db1a95afee1a5ee3a5a4b94d61b4dbd828387690de15ab0010868a424b8e WatchSource:0}: Error finding container e247db1a95afee1a5ee3a5a4b94d61b4dbd828387690de15ab0010868a424b8e: Status 404 returned error can't find the container with id e247db1a95afee1a5ee3a5a4b94d61b4dbd828387690de15ab0010868a424b8e
Nov 03 13:16:46.180942 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:46.180904    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:16:46.181060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:46.180996    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:16:46.181060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:46.181024    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:16:46.220489 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:46.220453    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" event={"ID":"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439","Type":"ContainerStarted","Data":"e247db1a95afee1a5ee3a5a4b94d61b4dbd828387690de15ab0010868a424b8e"}
Nov 03 13:16:47.238445 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:47.238409    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" event={"ID":"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439","Type":"ContainerStarted","Data":"d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7"}
Nov 03 13:16:47.355321 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:47.355272    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podStartSLOduration=2.355258842 podStartE2EDuration="2.355258842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:47.353597369 +0000 UTC m=+5148.925181311" watchObservedRunningTime="2026-11-03 13:16:47.355258842 +0000 UTC m=+5148.926842753"
Nov 03 13:16:47.739971 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:47.739937    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:16:47.740404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:47.740376    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Nov 03 13:16:47.740568 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:47.740544    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Nov 03 13:16:48.735776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:48.733354    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:48.735776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:48.733454    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:48.736606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:48.736581    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Nov 03 13:16:48.736851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:48.736829    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Nov 03 13:16:49.736090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:49.736053    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Nov 03 13:16:49.736603 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:49.736110    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Nov 03 13:16:50.736578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:50.736534    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Nov 03 13:16:50.737158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:50.736616    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Nov 03 13:16:51.736157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:51.736121    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Nov 03 13:16:51.736377 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:51.736192    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Nov 03 13:16:52.736338 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:52.736302    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Nov 03 13:16:52.736730 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:52.736356    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Nov 03 13:16:54.513449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.513412    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"]
Nov 03 13:16:54.514052 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.513672    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" containerID="cri-o://d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7" gracePeriod=30
Nov 03 13:16:54.670021 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.669971    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 13:16:54.727695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.727666    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume\") pod \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") "
Nov 03 13:16:54.727910 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.727725    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9\" (UniqueName: \"kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9\") pod \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\" (UID: \"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d\") "
Nov 03 13:16:54.728539 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.728510    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume" pod "cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" (UID: "cd4c2507-6e9f-4043-813c-a90c4aa1cd5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:54.731743 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.731717    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9" pod "cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" (UID: "cd4c2507-6e9f-4043-813c-a90c4aa1cd5d"). InnerVolumeSpecName "kube-api-access-dl7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:54.736019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.735982    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:16:54.736138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.736053    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:16:54.736882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.736862    4546 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 13:16:54.737025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.736902    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:16:54.742271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.742251    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 13:16:54.829494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.829455    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68\" (UniqueName: \"kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68\") pod \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") "
Nov 03 13:16:54.829694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.829662    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume\") pod \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\" (UID: \"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6\") "
Nov 03 13:16:54.830181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.830155    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:54.830324 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.830185    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7b9\" (UniqueName: \"kubernetes.io/projected/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d-kube-api-access-dl7b9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:54.830324 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.830151    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume" pod "bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" (UID: "bc11dafd-0e40-4c14-9bd6-9d6880b94bb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:16:54.831869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.831844    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68" pod "bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" (UID: "bc11dafd-0e40-4c14-9bd6-9d6880b94bb6"). InnerVolumeSpecName "kube-api-access-kpp68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:54.931334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.931239    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:54.931334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:54.931269    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp68\" (UniqueName: \"kubernetes.io/projected/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6-kube-api-access-kpp68\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:55.299773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.299718    4546 generic.go:356] "Generic (PLEG): container finished" podID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerID="b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce" exitCode=0
Nov 03 13:16:55.299979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.299779    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"
Nov 03 13:16:55.299979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.299791    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" event={"ID":"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d","Type":"ContainerDied","Data":"b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce"}
Nov 03 13:16:55.299979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.299869    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk" event={"ID":"cd4c2507-6e9f-4043-813c-a90c4aa1cd5d","Type":"ContainerDied","Data":"df64af9217410856e9080d3bfcad72682b1d2edb3086e2d19da520689a14bb86"}
Nov 03 13:16:55.299979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.299899    4546 scope.go:122] "RemoveContainer" containerID="b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce"
Nov 03 13:16:55.301461 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.301435    4546 generic.go:356] "Generic (PLEG): container finished" podID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerID="ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee" exitCode=0
Nov 03 13:16:55.301610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.301480    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"
Nov 03 13:16:55.301610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.301508    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" event={"ID":"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6","Type":"ContainerDied","Data":"ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee"}
Nov 03 13:16:55.301610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.301570    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx" event={"ID":"bc11dafd-0e40-4c14-9bd6-9d6880b94bb6","Type":"ContainerDied","Data":"ebe37801e8b99179d0a19835f273c087c57516329f24b8f9f909cf7eb429dcca"}
Nov 03 13:16:55.318509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.318485    4546 scope.go:122] "RemoveContainer" containerID="b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce"
Nov 03 13:16:55.318822 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:55.318800    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce\": container with ID starting with b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce not found: ID does not exist" containerID="b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce"
Nov 03 13:16:55.318909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.318827    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce"} err="failed to get container status \"b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce\": rpc error: code = NotFound desc = could not find container \"b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce\": container with ID starting with b908f9df82010fef4c0f073e27ae5a677725b49ff4a0ca3d0b283c7539e8a8ce not found: ID does not exist"
Nov 03 13:16:55.318909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.318846    4546 scope.go:122] "RemoveContainer" containerID="ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee"
Nov 03 13:16:55.331404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.331384    4546 scope.go:122] "RemoveContainer" containerID="ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee"
Nov 03 13:16:55.331668 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:16:55.331650    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee\": container with ID starting with ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee not found: ID does not exist" containerID="ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee"
Nov 03 13:16:55.331720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.331674    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee"} err="failed to get container status \"ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee\": rpc error: code = NotFound desc = could not find container \"ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee\": container with ID starting with ffab244413afe98e425589c3d869902d90bff918fa8bc2d62261f29e2970e9ee not found: ID does not exist"
Nov 03 13:16:55.452606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.452575    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"]
Nov 03 13:16:55.459314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.459286    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e0615c-77b6f7b7dc-dd6dx"]
Nov 03 13:16:55.560998 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.560893    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"]
Nov 03 13:16:55.563949 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:55.563923    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e0615c-8475684cf9-l2dwk"]
Nov 03 13:16:56.572846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:56.572815    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" path="/var/lib/kubelet/pods/bc11dafd-0e40-4c14-9bd6-9d6880b94bb6/volumes"
Nov 03 13:16:56.573239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:56.573227    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" path="/var/lib/kubelet/pods/cd4c2507-6e9f-4043-813c-a90c4aa1cd5d/volumes"
Nov 03 13:16:59.981362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:16:59.981330    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:17:00.093014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.092939    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.092978    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093019    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093167    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093269    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:00.093360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093321    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093482 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093363    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093482 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093404    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093475    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093513    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket\") pod \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\" (UID: \"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439\") "
Nov 03 13:17:00.093725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093700    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:00.093984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093960    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:00.094131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.094047    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.094250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.094148    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.094250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093974    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:00.094250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.093986    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:00.095491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.095470    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:00.095839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.095806    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:00.095940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.095863    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:17:00.095940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.095898    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm" pod "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" (UID: "2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439"). InnerVolumeSpecName "kube-api-access-8vhbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:00.195199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195170    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.195199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195194    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.195199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195203    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.195434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195215    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.195434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195223    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhbm\" (UniqueName: \"kubernetes.io/projected/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-kube-api-access-8vhbm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.195434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195232    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.195434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.195240    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:00.336068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.336036    4546 generic.go:356] "Generic (PLEG): container finished" podID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerID="d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7" exitCode=0
Nov 03 13:17:00.336262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.336087    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"
Nov 03 13:17:00.336262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.336121    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" event={"ID":"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439","Type":"ContainerDied","Data":"d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7"}
Nov 03 13:17:00.336262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.336178    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659" event={"ID":"2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439","Type":"ContainerDied","Data":"e247db1a95afee1a5ee3a5a4b94d61b4dbd828387690de15ab0010868a424b8e"}
Nov 03 13:17:00.336262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.336201    4546 scope.go:122] "RemoveContainer" containerID="d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7"
Nov 03 13:17:00.346842 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.346825    4546 scope.go:122] "RemoveContainer" containerID="d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7"
Nov 03 13:17:00.347141 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:00.347124    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7\": container with ID starting with d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7 not found: ID does not exist" containerID="d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7"
Nov 03 13:17:00.347184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.347151    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7"} err="failed to get container status \"d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7\": rpc error: code = NotFound desc = could not find container \"d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7\": container with ID starting with d7bcfe6cbd6a74f9008e9bdb4afa9150773591e917b4a25596f61150c18a63d7 not found: ID does not exist"
Nov 03 13:17:00.497555 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.497518    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"]
Nov 03 13:17:00.502762 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.502726    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-zs659"]
Nov 03 13:17:00.572906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:00.572877    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" path="/var/lib/kubelet/pods/2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439/volumes"
Nov 03 13:17:02.504214 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:02.504183    4546 scope.go:122] "RemoveContainer" containerID="1df9011df9f5863aaea627fa9490d3f08a148a846642d27feaa4394fb837364c"
Nov 03 13:17:02.513667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:02.513647    4546 scope.go:122] "RemoveContainer" containerID="381e146514da1d008e978619e25f3e8402a15046077ae2c769852c02376f1d37"
Nov 03 13:17:07.652008 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:07.651923    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:17:10.697483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.697446    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"]
Nov 03 13:17:10.697993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.697963    4546 cpu_manager.go:414] "containerMap: removing container" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerName="payload-processing"
Nov 03 13:17:10.697993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.697977    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerName="payload-processing"
Nov 03 13:17:10.698126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698002    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy"
Nov 03 13:17:10.698126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698008    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy"
Nov 03 13:17:10.698126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698023    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerName="payload-pre-processing"
Nov 03 13:17:10.698126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698030    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerName="payload-pre-processing"
Nov 03 13:17:10.698126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698110    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fef1e8f-9dfa-4a15-ac7e-cf93e0a65439" containerName="istio-proxy"
Nov 03 13:17:10.698126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698125    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc11dafd-0e40-4c14-9bd6-9d6880b94bb6" containerName="payload-pre-processing"
Nov 03 13:17:10.698438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.698138    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4c2507-6e9f-4043-813c-a90c4aa1cd5d" containerName="payload-processing"
Nov 03 13:17:10.703101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.703082    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.752054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.752028    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:17:10.798138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.798105    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:17:10.798303 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.798186    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:17:10.798303 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.798189    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:17:10.801169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.801149    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"]
Nov 03 13:17:10.819219 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.819190    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.819389 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.819316    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.819389 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.819357    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clswd\" (UniqueName: \"kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.920584 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.920554    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.920729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.920614    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.920729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.920639    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clswd\" (UniqueName: \"kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.921258 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.921235    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.923145 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.923119    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:10.975031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:10.975000    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clswd\" (UniqueName: \"kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd\") pod \"maas-api-revoke-sub-8ed5b204-rwg25\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:11.016904 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:11.016881    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:11.386821 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:11.386801    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"]
Nov 03 13:17:11.388011 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:17:11.387976    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e28be8_9a9e_4ccb_b570_16afdb29590f.slice/crio-a337bc47e90474e5755b974e8f8a395b472ba5fd0d6a8805e5cc5a337b4902bb WatchSource:0}: Error finding container a337bc47e90474e5755b974e8f8a395b472ba5fd0d6a8805e5cc5a337b4902bb: Status 404 returned error can't find the container with id a337bc47e90474e5755b974e8f8a395b472ba5fd0d6a8805e5cc5a337b4902bb
Nov 03 13:17:11.408529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:11.408496    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25" event={"ID":"40e28be8-9a9e-4ccb-b570-16afdb29590f","Type":"ContainerStarted","Data":"a337bc47e90474e5755b974e8f8a395b472ba5fd0d6a8805e5cc5a337b4902bb"}
Nov 03 13:17:12.415905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:12.415880    4546 generic.go:356] "Generic (PLEG): container finished" podID="40e28be8-9a9e-4ccb-b570-16afdb29590f" containerID="de351e6150482abe7de7b70f7c268b0de9cfd88582587abe285b26625cb174d2" exitCode=0
Nov 03 13:17:12.416297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:12.415932    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25" event={"ID":"40e28be8-9a9e-4ccb-b570-16afdb29590f","Type":"ContainerDied","Data":"de351e6150482abe7de7b70f7c268b0de9cfd88582587abe285b26625cb174d2"}
Nov 03 13:17:13.047628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.047593    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"]
Nov 03 13:17:13.051393 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.051364    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b024024e-9n4hz"]
Nov 03 13:17:13.569096 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.569065    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:13.753492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.753464    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd\" (UniqueName: \"kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd\") pod \"40e28be8-9a9e-4ccb-b570-16afdb29590f\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") "
Nov 03 13:17:13.753682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.753573    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca\") pod \"40e28be8-9a9e-4ccb-b570-16afdb29590f\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") "
Nov 03 13:17:13.753682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.753609    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca\") pod \"40e28be8-9a9e-4ccb-b570-16afdb29590f\" (UID: \"40e28be8-9a9e-4ccb-b570-16afdb29590f\") "
Nov 03 13:17:13.754037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.754014    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca" pod "40e28be8-9a9e-4ccb-b570-16afdb29590f" (UID: "40e28be8-9a9e-4ccb-b570-16afdb29590f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:13.755744 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.755716    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd" pod "40e28be8-9a9e-4ccb-b570-16afdb29590f" (UID: "40e28be8-9a9e-4ccb-b570-16afdb29590f"). InnerVolumeSpecName "kube-api-access-clswd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:13.755906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.755823    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca" pod "40e28be8-9a9e-4ccb-b570-16afdb29590f" (UID: "40e28be8-9a9e-4ccb-b570-16afdb29590f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:13.855268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.855229    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:13.855268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.855261    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e28be8-9a9e-4ccb-b570-16afdb29590f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:13.855268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:13.855275    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clswd\" (UniqueName: \"kubernetes.io/projected/40e28be8-9a9e-4ccb-b570-16afdb29590f-kube-api-access-clswd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:14.428788 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:14.428734    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"
Nov 03 13:17:14.428788 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:14.428776    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25" event={"ID":"40e28be8-9a9e-4ccb-b570-16afdb29590f","Type":"ContainerDied","Data":"a337bc47e90474e5755b974e8f8a395b472ba5fd0d6a8805e5cc5a337b4902bb"}
Nov 03 13:17:14.429051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:14.428847    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a337bc47e90474e5755b974e8f8a395b472ba5fd0d6a8805e5cc5a337b4902bb"
Nov 03 13:17:14.573101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:14.573069    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373ee556-9550-482f-8806-71b0d7147ba1" path="/var/lib/kubelet/pods/373ee556-9550-482f-8806-71b0d7147ba1/volumes"
Nov 03 13:17:15.077555 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:15.077522    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"]
Nov 03 13:17:15.123343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:15.123312    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"]
Nov 03 13:17:19.131300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:19.131261    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"]
Nov 03 13:17:19.131775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:19.131492    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy" containerID="cri-o://a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb" gracePeriod=30
Nov 03 13:17:24.403883 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.403825    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 13:17:24.467901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.467868    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.467909    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.467966    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.467992    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468079    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468114    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468182    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468255    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468289    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl\") pod \"323fc1aa-b75a-4423-a181-5ffc6b801396\" (UID: \"323fc1aa-b75a-4423-a181-5ffc6b801396\") "
Nov 03 13:17:24.468403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468288    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:24.468403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468288    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:24.468588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468490    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:24.468588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468535    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:24.468588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468552    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:24.468787 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468734    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.468787 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468784    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.469102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468798    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.469102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468812    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.469102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.468827    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323fc1aa-b75a-4423-a181-5ffc6b801396-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.470344 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.470321    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:17:24.470459 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.470433    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:24.470533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.470467    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:24.470789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.470775    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl" pod "323fc1aa-b75a-4423-a181-5ffc6b801396" (UID: "323fc1aa-b75a-4423-a181-5ffc6b801396"). InnerVolumeSpecName "kube-api-access-wmtgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:24.488032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.488009    4546 generic.go:356] "Generic (PLEG): container finished" podID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerID="a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb" exitCode=0
Nov 03 13:17:24.488130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.488053    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" event={"ID":"323fc1aa-b75a-4423-a181-5ffc6b801396","Type":"ContainerDied","Data":"a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb"}
Nov 03 13:17:24.488130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.488063    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"
Nov 03 13:17:24.488130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.488093    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg" event={"ID":"323fc1aa-b75a-4423-a181-5ffc6b801396","Type":"ContainerDied","Data":"3c188ab613e1377c40d929d001afc1604f574b5857ae513ac522c42cf947e747"}
Nov 03 13:17:24.488130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.488115    4546 scope.go:122] "RemoveContainer" containerID="a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb"
Nov 03 13:17:24.497763 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.497736    4546 scope.go:122] "RemoveContainer" containerID="ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"
Nov 03 13:17:24.506705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.506689    4546 scope.go:122] "RemoveContainer" containerID="a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb"
Nov 03 13:17:24.506993 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:24.506975    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb\": container with ID starting with a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb not found: ID does not exist" containerID="a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb"
Nov 03 13:17:24.507047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.507002    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb"} err="failed to get container status \"a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb\": rpc error: code = NotFound desc = could not find container \"a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb\": container with ID starting with a84a3771880615c3b3c4ecb47d2f54c985d2593023fb254c46ab25462b30a0cb not found: ID does not exist"
Nov 03 13:17:24.507047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.507022    4546 scope.go:122] "RemoveContainer" containerID="ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"
Nov 03 13:17:24.507267 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:24.507251    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9\": container with ID starting with ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9 not found: ID does not exist" containerID="ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"
Nov 03 13:17:24.507305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.507273    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9"} err="failed to get container status \"ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9\": rpc error: code = NotFound desc = could not find container \"ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9\": container with ID starting with ebed09413a8dd3ebccb91541f0e5644cd761da63c4c2654277e0a33ce98a2ec9 not found: ID does not exist"
Nov 03 13:17:24.570139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.570110    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.570139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.570136    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.570327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.570149    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323fc1aa-b75a-4423-a181-5ffc6b801396-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.570327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.570158    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmtgl\" (UniqueName: \"kubernetes.io/projected/323fc1aa-b75a-4423-a181-5ffc6b801396-kube-api-access-wmtgl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:24.665170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.665082    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"]
Nov 03 13:17:24.674061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:24.674021    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-02f899-openshift-default-5756bb6949-6vwtg"]
Nov 03 13:17:26.572423 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:26.572390    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" path="/var/lib/kubelet/pods/323fc1aa-b75a-4423-a181-5ffc6b801396/volumes"
Nov 03 13:17:27.426197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426165    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"]
Nov 03 13:17:27.426629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426617    4546 cpu_manager.go:414] "containerMap: removing container" podUID="40e28be8-9a9e-4ccb-b570-16afdb29590f" containerName="revoke-keys"
Nov 03 13:17:27.426675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426630    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e28be8-9a9e-4ccb-b570-16afdb29590f" containerName="revoke-keys"
Nov 03 13:17:27.426675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426653    4546 cpu_manager.go:414] "containerMap: removing container" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy"
Nov 03 13:17:27.426675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426659    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy"
Nov 03 13:17:27.426781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426676    4546 cpu_manager.go:414] "containerMap: removing container" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy"
Nov 03 13:17:27.426781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426682    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy"
Nov 03 13:17:27.426781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426765    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy"
Nov 03 13:17:27.426781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426775    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="323fc1aa-b75a-4423-a181-5ffc6b801396" containerName="istio-proxy"
Nov 03 13:17:27.426951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.426784    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e28be8-9a9e-4ccb-b570-16afdb29590f" containerName="revoke-keys"
Nov 03 13:17:27.431761 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.431730    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.522407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.509548    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwk28\" (UniqueName: \"kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.522407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.509714    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.522407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.514863    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"]
Nov 03 13:17:27.522903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.522876    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.529913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.529884    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:17:27.530141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.529913    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:17:27.530261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.529916    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:17:27.530261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.529954    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:17:27.624248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.624221    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwk28\" (UniqueName: \"kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.624712 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.624326    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.624712 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.624355    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.625086 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.625064    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.626857 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.626841    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.679797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.679712    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwk28\" (UniqueName: \"kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28\") pod \"maas-api-revoke-sub-0b7cdc9a-prjb8\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:27.744316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:27.744292    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:28.121234 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:28.121207    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"]
Nov 03 13:17:28.122457 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:17:28.122430    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0307d050_38d3_495d_948a_413773d4caed.slice/crio-969ba766c1bb7f148b01a4013bef1b345771dcc609da399f0312625554908b0e WatchSource:0}: Error finding container 969ba766c1bb7f148b01a4013bef1b345771dcc609da399f0312625554908b0e: Status 404 returned error can't find the container with id 969ba766c1bb7f148b01a4013bef1b345771dcc609da399f0312625554908b0e
Nov 03 13:17:28.416514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:28.416486    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"]
Nov 03 13:17:28.514923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:28.514884    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"]
Nov 03 13:17:28.538812 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:28.538734    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8" event={"ID":"0307d050-38d3-495d-948a-413773d4caed","Type":"ContainerStarted","Data":"2ec06588fb10db0e0cbb034225cd762a9af57c5814f2244802bccb53197e5c42"}
Nov 03 13:17:28.538812 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:28.538797    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8" event={"ID":"0307d050-38d3-495d-948a-413773d4caed","Type":"ContainerStarted","Data":"969ba766c1bb7f148b01a4013bef1b345771dcc609da399f0312625554908b0e"}
Nov 03 13:17:28.745236 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:28.745187    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8" podStartSLOduration=1.745168845 podStartE2EDuration="1.745168845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:28.742113993 +0000 UTC m=+5190.313697925" watchObservedRunningTime="2026-11-03 13:17:28.745168845 +0000 UTC m=+5190.316752757"
Nov 03 13:17:29.546668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:29.546631    4546 generic.go:356] "Generic (PLEG): container finished" podID="0307d050-38d3-495d-948a-413773d4caed" containerID="2ec06588fb10db0e0cbb034225cd762a9af57c5814f2244802bccb53197e5c42" exitCode=0
Nov 03 13:17:29.546894 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:29.546713    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8" event={"ID":"0307d050-38d3-495d-948a-413773d4caed","Type":"ContainerDied","Data":"2ec06588fb10db0e0cbb034225cd762a9af57c5814f2244802bccb53197e5c42"}
Nov 03 13:17:30.098441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.098399    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main" containerID="cri-o://4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c" gracePeriod=45
Nov 03 13:17:30.144027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.143978    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main" containerID="cri-o://e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090" gracePeriod=45
Nov 03 13:17:30.407355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.407328    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 13:17:30.439963 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.439941    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 13:17:30.555799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555453    4546 generic.go:356] "Generic (PLEG): container finished" podID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerID="4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c" exitCode=0
Nov 03 13:17:30.555799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555518    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"
Nov 03 13:17:30.555799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555572    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" event={"ID":"5bf7635f-08fc-42d4-b16f-81daee28d04c","Type":"ContainerDied","Data":"4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c"}
Nov 03 13:17:30.555799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555605    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52" event={"ID":"5bf7635f-08fc-42d4-b16f-81daee28d04c","Type":"ContainerDied","Data":"bdc73ede38f41d859011ccad3984309d7977890f72da2ac2ffce40ff5d9dde06"}
Nov 03 13:17:30.555799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555625    4546 scope.go:122] "RemoveContainer" containerID="4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c"
Nov 03 13:17:30.555799 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555723    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home\") pod \"5bf7635f-08fc-42d4-b16f-81daee28d04c\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555903    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache\") pod \"5bf7635f-08fc-42d4-b16f-81daee28d04c\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555950    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir\") pod \"5bf7635f-08fc-42d4-b16f-81daee28d04c\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.555990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir\") pod \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556025    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n\" (UniqueName: \"kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n\") pod \"5bf7635f-08fc-42d4-b16f-81daee28d04c\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556067    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm\") pod \"5bf7635f-08fc-42d4-b16f-81daee28d04c\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556108    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs\") pod \"5bf7635f-08fc-42d4-b16f-81daee28d04c\" (UID: \"5bf7635f-08fc-42d4-b16f-81daee28d04c\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556163    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs\") pod \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") "
Nov 03 13:17:30.556260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556190    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm\") pod \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") "
Nov 03 13:17:30.556654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556302    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache\") pod \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") "
Nov 03 13:17:30.556654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556343    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home\") pod \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") "
Nov 03 13:17:30.556654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556381    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4\" (UniqueName: \"kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4\") pod \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\" (UID: \"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6\") "
Nov 03 13:17:30.556654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556470    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir" pod "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" (UID: "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.556905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556682    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home" pod "5bf7635f-08fc-42d4-b16f-81daee28d04c" (UID: "5bf7635f-08fc-42d4-b16f-81daee28d04c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.556989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556904    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.556989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.556923    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.557155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.557132    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache" pod "5bf7635f-08fc-42d4-b16f-81daee28d04c" (UID: "5bf7635f-08fc-42d4-b16f-81daee28d04c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.557366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.557351    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir" pod "5bf7635f-08fc-42d4-b16f-81daee28d04c" (UID: "5bf7635f-08fc-42d4-b16f-81daee28d04c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.558251 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.558228    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache" pod "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" (UID: "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.558536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.558486    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home" pod "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" (UID: "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.560130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.559028    4546 generic.go:356] "Generic (PLEG): container finished" podID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerID="e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090" exitCode=0
Nov 03 13:17:30.560130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.559308    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"
Nov 03 13:17:30.560130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.559401    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" event={"ID":"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6","Type":"ContainerDied","Data":"e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090"}
Nov 03 13:17:30.560130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.559432    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf" event={"ID":"bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6","Type":"ContainerDied","Data":"f4c68abc922f80511ea7173f330dee19faed425dcb66a33fde17d582fbec9b8d"}
Nov 03 13:17:30.560722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.560700    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4" pod "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" (UID: "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6"). InnerVolumeSpecName "kube-api-access-lvpw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:30.561540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.561510    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm" pod "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" (UID: "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.561648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.561615    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs" pod "5bf7635f-08fc-42d4-b16f-81daee28d04c" (UID: "5bf7635f-08fc-42d4-b16f-81daee28d04c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:30.562162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.562133    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs" pod "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" (UID: "bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:30.562588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.562558    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm" pod "5bf7635f-08fc-42d4-b16f-81daee28d04c" (UID: "5bf7635f-08fc-42d4-b16f-81daee28d04c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:17:30.562975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.562952    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n" pod "5bf7635f-08fc-42d4-b16f-81daee28d04c" (UID: "5bf7635f-08fc-42d4-b16f-81daee28d04c"). InnerVolumeSpecName "kube-api-access-s642n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:30.575646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.575622    4546 scope.go:122] "RemoveContainer" containerID="4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c"
Nov 03 13:17:30.576013 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:30.575985    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c\": container with ID starting with 4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c not found: ID does not exist" containerID="4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c"
Nov 03 13:17:30.576121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.576021    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c"} err="failed to get container status \"4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c\": rpc error: code = NotFound desc = could not find container \"4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c\": container with ID starting with 4114a57cf77e2f8dd72f65b76230006edd56bec5cc80d455ee35967d7f71775c not found: ID does not exist"
Nov 03 13:17:30.576121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.576044    4546 scope.go:122] "RemoveContainer" containerID="e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090"
Nov 03 13:17:30.589730 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.589675    4546 scope.go:122] "RemoveContainer" containerID="e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090"
Nov 03 13:17:30.590058 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:30.590015    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090\": container with ID starting with e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090 not found: ID does not exist" containerID="e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090"
Nov 03 13:17:30.590168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.590067    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090"} err="failed to get container status \"e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090\": rpc error: code = NotFound desc = could not find container \"e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090\": container with ID starting with e7a7669b8bf7d1bb694b30ff49f73bd611db26e70b5452d0d92d4c1073a9d090 not found: ID does not exist"
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658010    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658039    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658068    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658083    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658098    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvpw4\" (UniqueName: \"kubernetes.io/projected/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6-kube-api-access-lvpw4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658116    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658131    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658145    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s642n\" (UniqueName: \"kubernetes.io/projected/5bf7635f-08fc-42d4-b16f-81daee28d04c-kube-api-access-s642n\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658161    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf7635f-08fc-42d4-b16f-81daee28d04c-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.658241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.658174    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf7635f-08fc-42d4-b16f-81daee28d04c-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.698820 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.698796    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:30.759186 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.759160    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28\" (UniqueName: \"kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28\") pod \"0307d050-38d3-495d-948a-413773d4caed\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") "
Nov 03 13:17:30.759346 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.759195    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca\") pod \"0307d050-38d3-495d-948a-413773d4caed\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") "
Nov 03 13:17:30.759346 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.759265    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca\") pod \"0307d050-38d3-495d-948a-413773d4caed\" (UID: \"0307d050-38d3-495d-948a-413773d4caed\") "
Nov 03 13:17:30.759602 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.759579    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca" pod "0307d050-38d3-495d-948a-413773d4caed" (UID: "0307d050-38d3-495d-948a-413773d4caed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:30.761412 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.761386    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca" pod "0307d050-38d3-495d-948a-413773d4caed" (UID: "0307d050-38d3-495d-948a-413773d4caed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:30.761514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.761458    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28" pod "0307d050-38d3-495d-948a-413773d4caed" (UID: "0307d050-38d3-495d-948a-413773d4caed"). InnerVolumeSpecName "kube-api-access-bwk28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:30.825499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.825464    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"]
Nov 03 13:17:30.860532 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.860498    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0307d050-38d3-495d-948a-413773d4caed-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.860685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.860535    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwk28\" (UniqueName: \"kubernetes.io/projected/0307d050-38d3-495d-948a-413773d4caed-kube-api-access-bwk28\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.860685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.860552    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0307d050-38d3-495d-948a-413773d4caed-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:30.926738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:30.926671    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"]
Nov 03 13:17:31.153665 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.153631    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"]
Nov 03 13:17:31.156482 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.156456    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/distinct-w2-91cd99-kserve-79d95d548d-jfz52"]
Nov 03 13:17:31.258998 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.258948    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"]
Nov 03 13:17:31.261868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.261841    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91cd99/simulator-w2-91cd99-kserve-6c94dbc6cb-k44xf"]
Nov 03 13:17:31.568585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.568509    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"
Nov 03 13:17:31.568585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.568514    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8" event={"ID":"0307d050-38d3-495d-948a-413773d4caed","Type":"ContainerDied","Data":"969ba766c1bb7f148b01a4013bef1b345771dcc609da399f0312625554908b0e"}
Nov 03 13:17:31.568585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:31.568552    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969ba766c1bb7f148b01a4013bef1b345771dcc609da399f0312625554908b0e"
Nov 03 13:17:32.573798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:32.573769    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" path="/var/lib/kubelet/pods/5bf7635f-08fc-42d4-b16f-81daee28d04c/volumes"
Nov 03 13:17:32.574223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:32.574209    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" path="/var/lib/kubelet/pods/bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6/volumes"
Nov 03 13:17:33.417018 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:33.416975    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerName="payload-processing" containerID="cri-o://f57550c763ff0aa47a8d6021340a3a3929be096d2c38af436107f063121873ad" gracePeriod=25
Nov 03 13:17:33.515376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:33.515323    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerName="payload-pre-processing" containerID="cri-o://bc18bebbea1b12a027545f201ddeede78f53c287a17caf99c5dc60552a2b8927" gracePeriod=25
Nov 03 13:17:35.826910 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:35.826868    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing" containerID="cri-o://1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0" gracePeriod=25
Nov 03 13:17:35.927487 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:35.927419    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerName="payload-pre-processing" containerID="cri-o://30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271" gracePeriod=25
Nov 03 13:17:36.006741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:36.006707    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:37.650188 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:37.650152    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:17:39.130392 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:39.130346    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:39.130882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:39.130351    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:45.001888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:45.001844    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:46.006687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:46.006649    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:49.130426 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:49.130382    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:49.130844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:49.130452    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:53.701971 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.701937    4546 generic.go:356] "Generic (PLEG): container finished" podID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerID="f57550c763ff0aa47a8d6021340a3a3929be096d2c38af436107f063121873ad" exitCode=0
Nov 03 13:17:53.702394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.702019    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" event={"ID":"72e6c6a9-aef7-4848-8e9e-da053fa51568","Type":"ContainerDied","Data":"f57550c763ff0aa47a8d6021340a3a3929be096d2c38af436107f063121873ad"}
Nov 03 13:17:53.702394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.702108    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt" event={"ID":"72e6c6a9-aef7-4848-8e9e-da053fa51568","Type":"ContainerDied","Data":"91ebb335b4e1a35e1f44278f8e1d426810210454be021114d7f6d5056f85cddd"}
Nov 03 13:17:53.702394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.702135    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ebb335b4e1a35e1f44278f8e1d426810210454be021114d7f6d5056f85cddd"
Nov 03 13:17:53.702394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.702345    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 13:17:53.704010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.703989    4546 generic.go:356] "Generic (PLEG): container finished" podID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerID="bc18bebbea1b12a027545f201ddeede78f53c287a17caf99c5dc60552a2b8927" exitCode=0
Nov 03 13:17:53.704116 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.704059    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" event={"ID":"057a30fe-c087-455a-9c93-68e38ed71dfb","Type":"ContainerDied","Data":"bc18bebbea1b12a027545f201ddeede78f53c287a17caf99c5dc60552a2b8927"}
Nov 03 13:17:53.761208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.761181    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 13:17:53.811876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.811850    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs\" (UniqueName: \"kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs\") pod \"72e6c6a9-aef7-4848-8e9e-da053fa51568\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") "
Nov 03 13:17:53.812071 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.812022    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg\" (UniqueName: \"kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg\") pod \"057a30fe-c087-455a-9c93-68e38ed71dfb\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") "
Nov 03 13:17:53.812071 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.812041    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume\") pod \"057a30fe-c087-455a-9c93-68e38ed71dfb\" (UID: \"057a30fe-c087-455a-9c93-68e38ed71dfb\") "
Nov 03 13:17:53.812204 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.812100    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume\") pod \"72e6c6a9-aef7-4848-8e9e-da053fa51568\" (UID: \"72e6c6a9-aef7-4848-8e9e-da053fa51568\") "
Nov 03 13:17:53.812664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.812626    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume" pod "057a30fe-c087-455a-9c93-68e38ed71dfb" (UID: "057a30fe-c087-455a-9c93-68e38ed71dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:53.812813 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.812678    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume" pod "72e6c6a9-aef7-4848-8e9e-da053fa51568" (UID: "72e6c6a9-aef7-4848-8e9e-da053fa51568"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:53.814794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.814771    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs" pod "72e6c6a9-aef7-4848-8e9e-da053fa51568" (UID: "72e6c6a9-aef7-4848-8e9e-da053fa51568"). InnerVolumeSpecName "kube-api-access-hv6gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:53.815257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.815234    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg" pod "057a30fe-c087-455a-9c93-68e38ed71dfb" (UID: "057a30fe-c087-455a-9c93-68e38ed71dfb"). InnerVolumeSpecName "kube-api-access-wbcdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:53.913168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.913132    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbcdg\" (UniqueName: \"kubernetes.io/projected/057a30fe-c087-455a-9c93-68e38ed71dfb-kube-api-access-wbcdg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:53.913168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.913168    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057a30fe-c087-455a-9c93-68e38ed71dfb-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:53.913442 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.913182    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e6c6a9-aef7-4848-8e9e-da053fa51568-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:53.913442 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:53.913190    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6gs\" (UniqueName: \"kubernetes.io/projected/72e6c6a9-aef7-4848-8e9e-da053fa51568-kube-api-access-hv6gs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:54.711125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:54.711043    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"
Nov 03 13:17:54.711125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:54.711080    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk" event={"ID":"057a30fe-c087-455a-9c93-68e38ed71dfb","Type":"ContainerDied","Data":"b3e10b4bd743cff4ed494a824603c7a579a23b6e9517e59458c54dfd5c28dd48"}
Nov 03 13:17:54.711773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:54.711147    4546 scope.go:122] "RemoveContainer" containerID="bc18bebbea1b12a027545f201ddeede78f53c287a17caf99c5dc60552a2b8927"
Nov 03 13:17:54.711773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:54.711248    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"
Nov 03 13:17:54.867761 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:54.867708    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"]
Nov 03 13:17:54.913505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:54.913457    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94e247-b899964f5-7nrgk"]
Nov 03 13:17:55.001948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:55.001912    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:55.014334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:55.014301    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"]
Nov 03 13:17:55.016964 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:55.016892    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94e247-84b5b9677f-xzpjt"]
Nov 03 13:17:55.752081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:55.752042    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"]
Nov 03 13:17:55.752612 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:55.752367    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" containerID="cri-o://c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf" gracePeriod=30
Nov 03 13:17:56.006568 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:17:56.006495    4546 logging.go:55] [core] [Channel #20150 SubChannel #20151]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.52:50052", ServerName: "10.133.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.52:50052: connect: connection refused"
Nov 03 13:17:56.105660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.105487    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 13:17:56.196479 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.196454    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 13:17:56.239076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.239043    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume\") pod \"376faac6-7b43-410b-bde9-1544dc445a93\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") "
Nov 03 13:17:56.239288 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.239140    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq\" (UniqueName: \"kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq\") pod \"376faac6-7b43-410b-bde9-1544dc445a93\" (UID: \"376faac6-7b43-410b-bde9-1544dc445a93\") "
Nov 03 13:17:56.239529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.239499    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume" pod "376faac6-7b43-410b-bde9-1544dc445a93" (UID: "376faac6-7b43-410b-bde9-1544dc445a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:56.241734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.241707    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq" pod "376faac6-7b43-410b-bde9-1544dc445a93" (UID: "376faac6-7b43-410b-bde9-1544dc445a93"). InnerVolumeSpecName "kube-api-access-kj4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:56.340641 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.340559    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb\" (UniqueName: \"kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb\") pod \"86cd0866-d324-4d00-8dd8-89c5590d815b\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") "
Nov 03 13:17:56.340641 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.340606    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume\") pod \"86cd0866-d324-4d00-8dd8-89c5590d815b\" (UID: \"86cd0866-d324-4d00-8dd8-89c5590d815b\") "
Nov 03 13:17:56.340996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.340975    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4hq\" (UniqueName: \"kubernetes.io/projected/376faac6-7b43-410b-bde9-1544dc445a93-kube-api-access-kj4hq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:56.340996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.340996    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376faac6-7b43-410b-bde9-1544dc445a93-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:56.341153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.341018    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume" pod "86cd0866-d324-4d00-8dd8-89c5590d815b" (UID: "86cd0866-d324-4d00-8dd8-89c5590d815b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:56.342984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.342910    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb" pod "86cd0866-d324-4d00-8dd8-89c5590d815b" (UID: "86cd0866-d324-4d00-8dd8-89c5590d815b"). InnerVolumeSpecName "kube-api-access-vmfcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:56.442397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.442359    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmfcb\" (UniqueName: \"kubernetes.io/projected/86cd0866-d324-4d00-8dd8-89c5590d815b-kube-api-access-vmfcb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:56.442397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.442389    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cd0866-d324-4d00-8dd8-89c5590d815b-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:56.572352 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.572318    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" path="/var/lib/kubelet/pods/057a30fe-c087-455a-9c93-68e38ed71dfb/volumes"
Nov 03 13:17:56.572714 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.572701    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" path="/var/lib/kubelet/pods/72e6c6a9-aef7-4848-8e9e-da053fa51568/volumes"
Nov 03 13:17:56.726498 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.726465    4546 generic.go:356] "Generic (PLEG): container finished" podID="376faac6-7b43-410b-bde9-1544dc445a93" containerID="1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0" exitCode=0
Nov 03 13:17:56.726706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.726514    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"
Nov 03 13:17:56.726706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.726549    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" event={"ID":"376faac6-7b43-410b-bde9-1544dc445a93","Type":"ContainerDied","Data":"1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0"}
Nov 03 13:17:56.726706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.726609    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" event={"ID":"376faac6-7b43-410b-bde9-1544dc445a93","Type":"ContainerDied","Data":"f4ba15c1fb1460c78318db712b8e25c0573c34f54736649e5352321d898f1da9"}
Nov 03 13:17:56.726706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.726644    4546 scope.go:122] "RemoveContainer" containerID="1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0"
Nov 03 13:17:56.727933 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.727915    4546 generic.go:356] "Generic (PLEG): container finished" podID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerID="30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271" exitCode=0
Nov 03 13:17:56.728025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.727960    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"
Nov 03 13:17:56.728025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.727986    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" event={"ID":"86cd0866-d324-4d00-8dd8-89c5590d815b","Type":"ContainerDied","Data":"30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271"}
Nov 03 13:17:56.728121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.728039    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r" event={"ID":"86cd0866-d324-4d00-8dd8-89c5590d815b","Type":"ContainerDied","Data":"1afc776f38db083d917c9a7343db593beea17c7894a66ab5e595bc3ef7bd2d46"}
Nov 03 13:17:56.736164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.736145    4546 scope.go:122] "RemoveContainer" containerID="1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0"
Nov 03 13:17:56.736436 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:56.736412    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0\": container with ID starting with 1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0 not found: ID does not exist" containerID="1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0"
Nov 03 13:17:56.736497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.736443    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0"} err="failed to get container status \"1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0\": rpc error: code = NotFound desc = could not find container \"1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0\": container with ID starting with 1892a050dbdf6f633f4cc00c46f2a460c09518b877233fc34a23ee2010da75f0 not found: ID does not exist"
Nov 03 13:17:56.736497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.736460    4546 scope.go:122] "RemoveContainer" containerID="30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271"
Nov 03 13:17:56.745010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.744984    4546 scope.go:122] "RemoveContainer" containerID="30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271"
Nov 03 13:17:56.745287 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:17:56.745270    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271\": container with ID starting with 30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271 not found: ID does not exist" containerID="30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271"
Nov 03 13:17:56.745335 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.745299    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271"} err="failed to get container status \"30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271\": rpc error: code = NotFound desc = could not find container \"30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271\": container with ID starting with 30dd50c7460100dbab221bcfc5ea810ef441574d7be0119855ec12363c045271 not found: ID does not exist"
Nov 03 13:17:56.879006 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.878975    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"]
Nov 03 13:17:56.881543 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.881517    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91cd99-d4f48d5c5-5l55r"]
Nov 03 13:17:56.983959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.983870    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"]
Nov 03 13:17:56.986453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:56.986424    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp"]
Nov 03 13:17:57.006763 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:57.006715    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-91cd99-5dcd84cb4f-sx5kp" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.52:50052\" within 1s: context deadline exceeded"
Nov 03 13:17:57.006949 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:17:57.006911    4546 logging.go:55] [core] [Channel #20150 SubChannel #20151]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.52:50052", ServerName: "10.133.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.52:50052: operation was canceled"
Nov 03 13:17:58.572298 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:58.572260    4546 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:17:58.572844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:58.572320    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:17:58.573056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:58.573029    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376faac6-7b43-410b-bde9-1544dc445a93" path="/var/lib/kubelet/pods/376faac6-7b43-410b-bde9-1544dc445a93/volumes"
Nov 03 13:17:58.573579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:17:58.573560    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" path="/var/lib/kubelet/pods/86cd0866-d324-4d00-8dd8-89c5590d815b/volumes"
Nov 03 13:18:00.997450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:00.997421    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 13:18:01.092491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092406    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.092491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092488    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.092777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092507    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.092777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092587    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.092777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092625    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.092777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092696    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.092777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092741    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.093087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092804    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.093087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092841    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy\") pod \"effffeda-ac8d-4c56-9cea-90eeac8842c8\" (UID: \"effffeda-ac8d-4c56-9cea-90eeac8842c8\") "
Nov 03 13:18:01.093087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.092841    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:01.093263 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093142    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:01.093338 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093314    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:01.093403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093378    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:01.093472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093422    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.093472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093445    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.093472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093459    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effffeda-ac8d-4c56-9cea-90eeac8842c8-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.093623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.093522    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:01.095149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.095122    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:18:01.095269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.095182    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:01.095269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.095245    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "kube-api-access-2zzdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:01.095483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.095467    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy" pod "effffeda-ac8d-4c56-9cea-90eeac8842c8" (UID: "effffeda-ac8d-4c56-9cea-90eeac8842c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:01.194944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.194909    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.194944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.194937    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.194944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.194948    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.195239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.194963    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.195239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.194977    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzdw\" (UniqueName: \"kubernetes.io/projected/effffeda-ac8d-4c56-9cea-90eeac8842c8-kube-api-access-2zzdw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.195239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.194987    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effffeda-ac8d-4c56-9cea-90eeac8842c8-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.765997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.765961    4546 generic.go:356] "Generic (PLEG): container finished" podID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerID="c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf" exitCode=0
Nov 03 13:18:01.766202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.766012    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"
Nov 03 13:18:01.766202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.766052    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" event={"ID":"effffeda-ac8d-4c56-9cea-90eeac8842c8","Type":"ContainerDied","Data":"c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf"}
Nov 03 13:18:01.766202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.766115    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj" event={"ID":"effffeda-ac8d-4c56-9cea-90eeac8842c8","Type":"ContainerDied","Data":"f3e2dd263ee6bfe7b7fc64c6c8c395540d71c32d807db3800db9f7add3e19972"}
Nov 03 13:18:01.766202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.766145    4546 scope.go:122] "RemoveContainer" containerID="c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf"
Nov 03 13:18:01.776559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.776540    4546 scope.go:122] "RemoveContainer" containerID="c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf"
Nov 03 13:18:01.776883 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:01.776860    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf\": container with ID starting with c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf not found: ID does not exist" containerID="c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf"
Nov 03 13:18:01.776950 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.776893    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf"} err="failed to get container status \"c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf\": rpc error: code = NotFound desc = could not find container \"c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf\": container with ID starting with c286ca30c319c77e96f2c5cfe98e64b7ccaab52f81a5b737a3f5213c24285fdf not found: ID does not exist"
Nov 03 13:18:01.918884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.918848    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"]
Nov 03 13:18:01.965093 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:01.965055    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-dbsgj"]
Nov 03 13:18:02.569194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:02.569166    4546 scope.go:122] "RemoveContainer" containerID="6c224c26cf85e9a4b6d2005b632f26dd847fc08b61c6007e8272f9ccefee6e4e"
Nov 03 13:18:02.572759 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:02.572723    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" path="/var/lib/kubelet/pods/effffeda-ac8d-4c56-9cea-90eeac8842c8/volumes"
Nov 03 13:18:02.578570 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:02.578552    4546 scope.go:122] "RemoveContainer" containerID="f57550c763ff0aa47a8d6021340a3a3929be096d2c38af436107f063121873ad"
Nov 03 13:18:07.652064 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:07.652029    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:18:12.530039 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:12.530000    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"]
Nov 03 13:18:12.530585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:12.530240    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy" containerID="cri-o://ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5" gracePeriod=30
Nov 03 13:18:13.603518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603478    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"]
Nov 03 13:18:13.603972 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603941    4546 cpu_manager.go:414] "containerMap: removing container" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing"
Nov 03 13:18:13.603972 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603954    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing"
Nov 03 13:18:13.603972 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603963    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0307d050-38d3-495d-948a-413773d4caed" containerName="revoke-keys"
Nov 03 13:18:13.603972 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603968    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0307d050-38d3-495d-948a-413773d4caed" containerName="revoke-keys"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603976    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603982    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603991    4546 cpu_manager.go:414] "containerMap: removing container" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerName="payload-pre-processing"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.603998    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerName="payload-pre-processing"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604006    4546 cpu_manager.go:414] "containerMap: removing container" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerName="payload-pre-processing"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604011    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerName="payload-pre-processing"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604042    4546 cpu_manager.go:414] "containerMap: removing container" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604046    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604056    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604061    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604078    4546 cpu_manager.go:414] "containerMap: removing container" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerName="payload-processing"
Nov 03 13:18:13.604115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604083    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerName="payload-processing"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604147    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e6c6a9-aef7-4848-8e9e-da053fa51568" containerName="payload-processing"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604157    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="376faac6-7b43-410b-bde9-1544dc445a93" containerName="payload-processing"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604162    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0307d050-38d3-495d-948a-413773d4caed" containerName="revoke-keys"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604167    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdf272c-f4ab-413c-b83d-9d6aacdd4bc6" containerName="main"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604174    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="057a30fe-c087-455a-9c93-68e38ed71dfb" containerName="payload-pre-processing"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604179    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="86cd0866-d324-4d00-8dd8-89c5590d815b" containerName="payload-pre-processing"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604184    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="effffeda-ac8d-4c56-9cea-90eeac8842c8" containerName="istio-proxy"
Nov 03 13:18:13.604548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.604191    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf7635f-08fc-42d4-b16f-81daee28d04c" containerName="main"
Nov 03 13:18:13.607518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.607489    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.625521 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.625488    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.625682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.625532    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszm6\" (UniqueName: \"kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.625816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.625787    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.658517 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.658482    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:18:13.658826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.658799    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:18:13.705036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.705007    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:18:13.705036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.705027    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:18:13.707858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.707837    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"]
Nov 03 13:18:13.726373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.726338    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.726566 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.726385    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszm6\" (UniqueName: \"kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.726566 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.726481    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.727093 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.727071    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.729073 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.729056    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.781647 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.781615    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszm6\" (UniqueName: \"kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6\") pod \"maas-api-revoke-sub-1c010f8e-j8sfc\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:13.921225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:13.921130    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:14.288505 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:14.288479    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"]
Nov 03 13:18:14.289905 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:18:14.289878    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72db8557_0ee4_4e3a_b6e9_86415129e9e4.slice/crio-861e2e1c02f02e991f645ea9d6140bb4776ba582eb399f6717a35cdf21349519 WatchSource:0}: Error finding container 861e2e1c02f02e991f645ea9d6140bb4776ba582eb399f6717a35cdf21349519: Status 404 returned error can't find the container with id 861e2e1c02f02e991f645ea9d6140bb4776ba582eb399f6717a35cdf21349519
Nov 03 13:18:14.838554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:14.838512    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc" event={"ID":"72db8557-0ee4-4e3a-b6e9-86415129e9e4","Type":"ContainerStarted","Data":"15a4420520c97d5e856b26c468b98288f4fee2c30483d26bb7efb2e55e763f5e"}
Nov 03 13:18:14.838979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:14.838563    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc" event={"ID":"72db8557-0ee4-4e3a-b6e9-86415129e9e4","Type":"ContainerStarted","Data":"861e2e1c02f02e991f645ea9d6140bb4776ba582eb399f6717a35cdf21349519"}
Nov 03 13:18:14.947190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:14.947085    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc" podStartSLOduration=1.947069586 podStartE2EDuration="1.947069586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:14.944569947 +0000 UTC m=+5236.516153857" watchObservedRunningTime="2026-11-03 13:18:14.947069586 +0000 UTC m=+5236.518653552"
Nov 03 13:18:15.845915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:15.845880    4546 generic.go:356] "Generic (PLEG): container finished" podID="72db8557-0ee4-4e3a-b6e9-86415129e9e4" containerID="15a4420520c97d5e856b26c468b98288f4fee2c30483d26bb7efb2e55e763f5e" exitCode=0
Nov 03 13:18:15.846337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:15.845961    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc" event={"ID":"72db8557-0ee4-4e3a-b6e9-86415129e9e4","Type":"ContainerDied","Data":"15a4420520c97d5e856b26c468b98288f4fee2c30483d26bb7efb2e55e763f5e"}
Nov 03 13:18:16.990102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:16.990075    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:17.060539 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.060501    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca\") pod \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") "
Nov 03 13:18:17.060731 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.060586    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca\") pod \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") "
Nov 03 13:18:17.060731 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.060638    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6\" (UniqueName: \"kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6\") pod \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\" (UID: \"72db8557-0ee4-4e3a-b6e9-86415129e9e4\") "
Nov 03 13:18:17.060896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.060874    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca" pod "72db8557-0ee4-4e3a-b6e9-86415129e9e4" (UID: "72db8557-0ee4-4e3a-b6e9-86415129e9e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:17.061000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.060984    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.062909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.062884    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca" pod "72db8557-0ee4-4e3a-b6e9-86415129e9e4" (UID: "72db8557-0ee4-4e3a-b6e9-86415129e9e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:17.063010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.062954    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6" pod "72db8557-0ee4-4e3a-b6e9-86415129e9e4" (UID: "72db8557-0ee4-4e3a-b6e9-86415129e9e4"). InnerVolumeSpecName "kube-api-access-dszm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:17.162187 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.162106    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72db8557-0ee4-4e3a-b6e9-86415129e9e4-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.162187 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.162133    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dszm6\" (UniqueName: \"kubernetes.io/projected/72db8557-0ee4-4e3a-b6e9-86415129e9e4-kube-api-access-dszm6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.768352 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.768323    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 13:18:17.858739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.858646    4546 generic.go:356] "Generic (PLEG): container finished" podID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerID="ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5" exitCode=0
Nov 03 13:18:17.858739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.858692    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"
Nov 03 13:18:17.859031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.858732    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" event={"ID":"9f552f2f-d534-499d-827c-093c8c4f1b22","Type":"ContainerDied","Data":"ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5"}
Nov 03 13:18:17.859031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.858822    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5" event={"ID":"9f552f2f-d534-499d-827c-093c8c4f1b22","Type":"ContainerDied","Data":"0bd50fd62337bc9e9346604ca91be0809b75337e1d83885cc033ceac04480861"}
Nov 03 13:18:17.859031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.858849    4546 scope.go:122] "RemoveContainer" containerID="ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5"
Nov 03 13:18:17.860278 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.860262    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"
Nov 03 13:18:17.860363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.860259    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc" event={"ID":"72db8557-0ee4-4e3a-b6e9-86415129e9e4","Type":"ContainerDied","Data":"861e2e1c02f02e991f645ea9d6140bb4776ba582eb399f6717a35cdf21349519"}
Nov 03 13:18:17.860409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.860386    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="861e2e1c02f02e991f645ea9d6140bb4776ba582eb399f6717a35cdf21349519"
Nov 03 13:18:17.867622 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867597    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.867716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867642    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.867716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867689    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.867850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867807    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.867895 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867868    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.867951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867903    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.867951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867920    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:17.867951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.867940    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.868123 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868010    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.868123 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868042    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert\") pod \"9f552f2f-d534-499d-827c-093c8c4f1b22\" (UID: \"9f552f2f-d534-499d-827c-093c8c4f1b22\") "
Nov 03 13:18:17.868123 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868063    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:17.868314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868165    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:17.868314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868256    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:17.868517 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868495    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.868833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868521    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.868833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868537    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.868833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868551    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.868833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.868803    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:17.870179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.870114    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:18:17.870179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.870146    4546 scope.go:122] "RemoveContainer" containerID="ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5"
Nov 03 13:18:17.870448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.870415    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:17.870547 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:17.870475    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5\": container with ID starting with ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5 not found: ID does not exist" containerID="ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5"
Nov 03 13:18:17.870547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.870499    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5"} err="failed to get container status \"ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5\": rpc error: code = NotFound desc = could not find container \"ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5\": container with ID starting with ead9f4be87af68d291689e4cea48e4d46b49e1e2191d89ddfc03c69b74bc10e5 not found: ID does not exist"
Nov 03 13:18:17.870634 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.870597    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "kube-api-access-bxcgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:17.870634 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.870607    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token" pod "9f552f2f-d534-499d-827c-093c8c4f1b22" (UID: "9f552f2f-d534-499d-827c-093c8c4f1b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:17.933156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.933125    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"]
Nov 03 13:18:17.969129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.969084    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxcgz\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-kube-api-access-bxcgz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.969129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.969119    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.969129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.969129    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.969129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.969141    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f552f2f-d534-499d-827c-093c8c4f1b22-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:17.969507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:17.969150    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f552f2f-d534-499d-827c-093c8c4f1b22-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:18.273402 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:18.273360    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"]
Nov 03 13:18:18.278522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:18.278489    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-rx5w5"]
Nov 03 13:18:18.572002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:18.571972    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" path="/var/lib/kubelet/pods/9f552f2f-d534-499d-827c-093c8c4f1b22/volumes"
Nov 03 13:18:28.702673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.702641    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"]
Nov 03 13:18:28.703197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.703075    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy"
Nov 03 13:18:28.703197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.703087    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy"
Nov 03 13:18:28.703197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.703103    4546 cpu_manager.go:414] "containerMap: removing container" podUID="72db8557-0ee4-4e3a-b6e9-86415129e9e4" containerName="revoke-keys"
Nov 03 13:18:28.703197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.703108    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72db8557-0ee4-4e3a-b6e9-86415129e9e4" containerName="revoke-keys"
Nov 03 13:18:28.703197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.703173    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="72db8557-0ee4-4e3a-b6e9-86415129e9e4" containerName="revoke-keys"
Nov 03 13:18:28.703197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.703188    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f552f2f-d534-499d-827c-093c8c4f1b22" containerName="istio-proxy"
Nov 03 13:18:28.707509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.707488    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.784664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.784629    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwf9d\" (UniqueName: \"kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.784664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.784671    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.784921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.784707    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.804961 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.804927    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:18:28.805104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.805000    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:18:28.805104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.805068    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:18:28.805104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.805086    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:18:28.807715 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.807687    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"]
Nov 03 13:18:28.886225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.886197    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.886400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.886243    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.886400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.886329    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwf9d\" (UniqueName: \"kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.886950 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.886928    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.888887 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.888866    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:28.940590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:28.940556    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwf9d\" (UniqueName: \"kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d\") pod \"maas-api-revoke-sub-bc27b28e-zghpm\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:29.020383 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:29.020352    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:29.127424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:29.127390    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"]
Nov 03 13:18:29.389476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:29.389440    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"]
Nov 03 13:18:29.391088 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:18:29.391052    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e76445_d308_4ee2_8320_64d038044c41.slice/crio-523d2207493fd0c30977c81c3f2d06050eab1d147e213eed47abe4dd14860fb8 WatchSource:0}: Error finding container 523d2207493fd0c30977c81c3f2d06050eab1d147e213eed47abe4dd14860fb8: Status 404 returned error can't find the container with id 523d2207493fd0c30977c81c3f2d06050eab1d147e213eed47abe4dd14860fb8
Nov 03 13:18:29.926474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:29.926442    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm" event={"ID":"07e76445-d308-4ee2-8320-64d038044c41","Type":"ContainerStarted","Data":"fdde23b4ae5b55216f1b2afa6f95189dc28c04dc3cbab2c23f05354afbe7fa8b"}
Nov 03 13:18:29.927042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:29.926504    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm" event={"ID":"07e76445-d308-4ee2-8320-64d038044c41","Type":"ContainerStarted","Data":"523d2207493fd0c30977c81c3f2d06050eab1d147e213eed47abe4dd14860fb8"}
Nov 03 13:18:30.034397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:30.034291    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm" podStartSLOduration=2.034276999 podStartE2EDuration="2.034276999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:30.032282527 +0000 UTC m=+5251.603866440" watchObservedRunningTime="2026-11-03 13:18:30.034276999 +0000 UTC m=+5251.605860938"
Nov 03 13:18:30.934346 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:30.934314    4546 generic.go:356] "Generic (PLEG): container finished" podID="07e76445-d308-4ee2-8320-64d038044c41" containerID="fdde23b4ae5b55216f1b2afa6f95189dc28c04dc3cbab2c23f05354afbe7fa8b" exitCode=0
Nov 03 13:18:30.934813 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:30.934390    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm" event={"ID":"07e76445-d308-4ee2-8320-64d038044c41","Type":"ContainerDied","Data":"fdde23b4ae5b55216f1b2afa6f95189dc28c04dc3cbab2c23f05354afbe7fa8b"}
Nov 03 13:18:32.087063 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.087034    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:32.220276 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.220227    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d\" (UniqueName: \"kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d\") pod \"07e76445-d308-4ee2-8320-64d038044c41\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") "
Nov 03 13:18:32.220495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.220451    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca\") pod \"07e76445-d308-4ee2-8320-64d038044c41\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") "
Nov 03 13:18:32.220624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.220599    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca\") pod \"07e76445-d308-4ee2-8320-64d038044c41\" (UID: \"07e76445-d308-4ee2-8320-64d038044c41\") "
Nov 03 13:18:32.221064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.221035    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca" pod "07e76445-d308-4ee2-8320-64d038044c41" (UID: "07e76445-d308-4ee2-8320-64d038044c41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:32.222839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.222807    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca" pod "07e76445-d308-4ee2-8320-64d038044c41" (UID: "07e76445-d308-4ee2-8320-64d038044c41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:32.223131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.223109    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d" pod "07e76445-d308-4ee2-8320-64d038044c41" (UID: "07e76445-d308-4ee2-8320-64d038044c41"). InnerVolumeSpecName "kube-api-access-mwf9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:32.326280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.323154    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e76445-d308-4ee2-8320-64d038044c41-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.326280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.323201    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e76445-d308-4ee2-8320-64d038044c41-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.326280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.323220    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwf9d\" (UniqueName: \"kubernetes.io/projected/07e76445-d308-4ee2-8320-64d038044c41-kube-api-access-mwf9d\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:32.947707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.947614    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"
Nov 03 13:18:32.947928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.947618    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm" event={"ID":"07e76445-d308-4ee2-8320-64d038044c41","Type":"ContainerDied","Data":"523d2207493fd0c30977c81c3f2d06050eab1d147e213eed47abe4dd14860fb8"}
Nov 03 13:18:32.947928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.947777    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="523d2207493fd0c30977c81c3f2d06050eab1d147e213eed47abe4dd14860fb8"
Nov 03 13:18:32.954247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:32.954199    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" containerID="cri-o://f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131" gracePeriod=45
Nov 03 13:18:33.219912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.219886    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 13:18:33.332273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332242    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir\") pod \"a64e6e11-031b-46be-a282-657483c47b7a\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") "
Nov 03 13:18:33.332483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332356    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm\") pod \"a64e6e11-031b-46be-a282-657483c47b7a\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") "
Nov 03 13:18:33.332483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332384    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache\") pod \"a64e6e11-031b-46be-a282-657483c47b7a\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") "
Nov 03 13:18:33.332483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332420    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home\") pod \"a64e6e11-031b-46be-a282-657483c47b7a\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") "
Nov 03 13:18:33.332698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332491    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f\") pod \"a64e6e11-031b-46be-a282-657483c47b7a\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") "
Nov 03 13:18:33.332698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332553    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs\") pod \"a64e6e11-031b-46be-a282-657483c47b7a\" (UID: \"a64e6e11-031b-46be-a282-657483c47b7a\") "
Nov 03 13:18:33.335339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.332943    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir" pod "a64e6e11-031b-46be-a282-657483c47b7a" (UID: "a64e6e11-031b-46be-a282-657483c47b7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.335339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.333057    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home" pod "a64e6e11-031b-46be-a282-657483c47b7a" (UID: "a64e6e11-031b-46be-a282-657483c47b7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.335339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.333186    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache" pod "a64e6e11-031b-46be-a282-657483c47b7a" (UID: "a64e6e11-031b-46be-a282-657483c47b7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.335737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.335695    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs" pod "a64e6e11-031b-46be-a282-657483c47b7a" (UID: "a64e6e11-031b-46be-a282-657483c47b7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:33.335878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.335799    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f" pod "a64e6e11-031b-46be-a282-657483c47b7a" (UID: "a64e6e11-031b-46be-a282-657483c47b7a"). InnerVolumeSpecName "kube-api-access-92c8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:33.336012 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.335988    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm" pod "a64e6e11-031b-46be-a282-657483c47b7a" (UID: "a64e6e11-031b-46be-a282-657483c47b7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.433953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.433907    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.433953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.433939    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.433953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.433948    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.433953 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.433958    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64e6e11-031b-46be-a282-657483c47b7a-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.434322 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.433970    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/a64e6e11-031b-46be-a282-657483c47b7a-kube-api-access-92c8f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.434322 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.433979    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64e6e11-031b-46be-a282-657483c47b7a-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.957366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.957320    4546 generic.go:356] "Generic (PLEG): container finished" podID="a64e6e11-031b-46be-a282-657483c47b7a" containerID="f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131" exitCode=0
Nov 03 13:18:33.957609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.957375    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" event={"ID":"a64e6e11-031b-46be-a282-657483c47b7a","Type":"ContainerDied","Data":"f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131"}
Nov 03 13:18:33.957609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.957419    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" event={"ID":"a64e6e11-031b-46be-a282-657483c47b7a","Type":"ContainerDied","Data":"1c5f70a199ade095170bb8661515056fe0e400170ccdec7f38fc0e3f3e02d15a"}
Nov 03 13:18:33.957609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.957425    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"
Nov 03 13:18:33.957609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.957442    4546 scope.go:122] "RemoveContainer" containerID="f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131"
Nov 03 13:18:33.971219 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.971195    4546 scope.go:122] "RemoveContainer" containerID="f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131"
Nov 03 13:18:33.971605 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:33.971581    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131\": container with ID starting with f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131 not found: ID does not exist" containerID="f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131"
Nov 03 13:18:33.971692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:33.971618    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131"} err="failed to get container status \"f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131\": rpc error: code = NotFound desc = could not find container \"f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131\": container with ID starting with f8a75b8857039dbb481a1d2b75060b860f855bc885b6b8949f979cc3eced7131 not found: ID does not exist"
Nov 03 13:18:34.113758 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.113703    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"]
Nov 03 13:18:34.128329 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.128275    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v" podUID="7207c66b-8ae9-491f-b723-9cbfe619fca6" containerName="payload-processing" containerID="cri-o://c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f" gracePeriod=25
Nov 03 13:18:34.159593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.159563    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d"]
Nov 03 13:18:34.170807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.170760    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/distinct-2-w1-5951bc-kserve-cf84d5f5b-nxt2d" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:18:34.387465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.387438    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 13:18:34.445180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.445147    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume\") pod \"7207c66b-8ae9-491f-b723-9cbfe619fca6\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") "
Nov 03 13:18:34.445406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.445244    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28\" (UniqueName: \"kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28\") pod \"7207c66b-8ae9-491f-b723-9cbfe619fca6\" (UID: \"7207c66b-8ae9-491f-b723-9cbfe619fca6\") "
Nov 03 13:18:34.445622 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.445599    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume" pod "7207c66b-8ae9-491f-b723-9cbfe619fca6" (UID: "7207c66b-8ae9-491f-b723-9cbfe619fca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:34.447532 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.447498    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28" pod "7207c66b-8ae9-491f-b723-9cbfe619fca6" (UID: "7207c66b-8ae9-491f-b723-9cbfe619fca6"). InnerVolumeSpecName "kube-api-access-2bg28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:34.546418 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.546319    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7207c66b-8ae9-491f-b723-9cbfe619fca6-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:34.546418 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.546356    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bg28\" (UniqueName: \"kubernetes.io/projected/7207c66b-8ae9-491f-b723-9cbfe619fca6-kube-api-access-2bg28\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:34.572373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.572330    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64e6e11-031b-46be-a282-657483c47b7a" path="/var/lib/kubelet/pods/a64e6e11-031b-46be-a282-657483c47b7a/volumes"
Nov 03 13:18:34.963822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.963716    4546 generic.go:356] "Generic (PLEG): container finished" podID="7207c66b-8ae9-491f-b723-9cbfe619fca6" containerID="c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f" exitCode=0
Nov 03 13:18:34.963822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.963799    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"
Nov 03 13:18:34.964087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.963808    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v" event={"ID":"7207c66b-8ae9-491f-b723-9cbfe619fca6","Type":"ContainerDied","Data":"c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f"}
Nov 03 13:18:34.964087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.963884    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v" event={"ID":"7207c66b-8ae9-491f-b723-9cbfe619fca6","Type":"ContainerDied","Data":"99697b07294e81c33f557b298f4204a4d215435bfc1e50f0d72d191be1f79184"}
Nov 03 13:18:34.964087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.963913    4546 scope.go:122] "RemoveContainer" containerID="c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f"
Nov 03 13:18:34.973170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.973147    4546 scope.go:122] "RemoveContainer" containerID="c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f"
Nov 03 13:18:34.973509 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:34.973488    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f\": container with ID starting with c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f not found: ID does not exist" containerID="c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f"
Nov 03 13:18:34.973583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:34.973520    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f"} err="failed to get container status \"c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f\": rpc error: code = NotFound desc = could not find container \"c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f\": container with ID starting with c09066496e90c2954eb3cadab8fa74fe3bd99ba5e4120c8cb9276bdef983a86f not found: ID does not exist"
Nov 03 13:18:35.116352 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:35.116310    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"]
Nov 03 13:18:35.118965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:35.118934    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5951bc-6cc6b6c5c5-vhz7v"]
Nov 03 13:18:35.715425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:35.715387    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"]
Nov 03 13:18:36.572198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:36.572164    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7207c66b-8ae9-491f-b723-9cbfe619fca6" path="/var/lib/kubelet/pods/7207c66b-8ae9-491f-b723-9cbfe619fca6/volumes"
Nov 03 13:18:37.650827 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:37.650735    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:18:40.716131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:40.716084    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerName="payload-processing" containerID="cri-o://6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273" gracePeriod=25
Nov 03 13:18:46.481229 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:46.481189    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:48.198333 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:48.198294    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"]
Nov 03 13:18:48.198866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:48.198528    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy" containerID="cri-o://4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372" gracePeriod=30
Nov 03 13:18:49.265891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.265851    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"]
Nov 03 13:18:49.266288 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266269    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7207c66b-8ae9-491f-b723-9cbfe619fca6" containerName="payload-processing"
Nov 03 13:18:49.266288 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266280    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7207c66b-8ae9-491f-b723-9cbfe619fca6" containerName="payload-processing"
Nov 03 13:18:49.266357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266294    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main"
Nov 03 13:18:49.266357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266301    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main"
Nov 03 13:18:49.266357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266312    4546 cpu_manager.go:414] "containerMap: removing container" podUID="07e76445-d308-4ee2-8320-64d038044c41" containerName="revoke-keys"
Nov 03 13:18:49.266357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266318    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e76445-d308-4ee2-8320-64d038044c41" containerName="revoke-keys"
Nov 03 13:18:49.266512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266392    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7207c66b-8ae9-491f-b723-9cbfe619fca6" containerName="payload-processing"
Nov 03 13:18:49.266512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266402    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64e6e11-031b-46be-a282-657483c47b7a" containerName="main"
Nov 03 13:18:49.266512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.266409    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e76445-d308-4ee2-8320-64d038044c41" containerName="revoke-keys"
Nov 03 13:18:49.269541 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.269520    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.321077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.321009    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:18:49.321077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.321007    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:18:49.323827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.323779    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"]
Nov 03 13:18:49.366094 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.366063    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:18:49.366094 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.366063    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:18:49.403300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.403257    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.403510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.403341    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.403510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.403466    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l424k\" (UniqueName: \"kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.504492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.504455    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.504707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.504536    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l424k\" (UniqueName: \"kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.504707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.504600    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.505282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.505250    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.507070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.507044    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.559018 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.558939    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l424k\" (UniqueName: \"kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k\") pod \"maas-api-revoke-sub-50fcaaff-mknxl\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.583204 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.583175    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:49.950163 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:49.950131    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"]
Nov 03 13:18:49.952156 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:18:49.952122    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd477a5_35e1_4ed9_a51a_2900c170431f.slice/crio-bc7fea63bae94d996f4203ff612be0e68f98d5c089e5b461c05e3e4813144a1c WatchSource:0}: Error finding container bc7fea63bae94d996f4203ff612be0e68f98d5c089e5b461c05e3e4813144a1c: Status 404 returned error can't find the container with id bc7fea63bae94d996f4203ff612be0e68f98d5c089e5b461c05e3e4813144a1c
Nov 03 13:18:50.050637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:50.050598    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl" event={"ID":"bcd477a5-35e1-4ed9-a51a-2900c170431f","Type":"ContainerStarted","Data":"bc7fea63bae94d996f4203ff612be0e68f98d5c089e5b461c05e3e4813144a1c"}
Nov 03 13:18:51.044969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:51.044877    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"]
Nov 03 13:18:51.058313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:51.058289    4546 generic.go:356] "Generic (PLEG): container finished" podID="bcd477a5-35e1-4ed9-a51a-2900c170431f" containerID="35e91131ce57228783d80e6a4193fc52e823fa604b8d7aaf7bf3252ca3b652a3" exitCode=0
Nov 03 13:18:51.058452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:51.058384    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl" event={"ID":"bcd477a5-35e1-4ed9-a51a-2900c170431f","Type":"ContainerDied","Data":"35e91131ce57228783d80e6a4193fc52e823fa604b8d7aaf7bf3252ca3b652a3"}
Nov 03 13:18:51.091091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:51.091057    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def83461-cm8h4"]
Nov 03 13:18:52.212369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.212343    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:52.334286 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.334203    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k\" (UniqueName: \"kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k\") pod \"bcd477a5-35e1-4ed9-a51a-2900c170431f\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") "
Nov 03 13:18:52.334467 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.334298    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca\") pod \"bcd477a5-35e1-4ed9-a51a-2900c170431f\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") "
Nov 03 13:18:52.334467 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.334401    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca\") pod \"bcd477a5-35e1-4ed9-a51a-2900c170431f\" (UID: \"bcd477a5-35e1-4ed9-a51a-2900c170431f\") "
Nov 03 13:18:52.334766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.334719    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca" pod "bcd477a5-35e1-4ed9-a51a-2900c170431f" (UID: "bcd477a5-35e1-4ed9-a51a-2900c170431f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:52.336562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.336530    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca" pod "bcd477a5-35e1-4ed9-a51a-2900c170431f" (UID: "bcd477a5-35e1-4ed9-a51a-2900c170431f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:52.336673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.336625    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k" pod "bcd477a5-35e1-4ed9-a51a-2900c170431f" (UID: "bcd477a5-35e1-4ed9-a51a-2900c170431f"). InnerVolumeSpecName "kube-api-access-l424k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:52.435646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.435595    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l424k\" (UniqueName: \"kubernetes.io/projected/bcd477a5-35e1-4ed9-a51a-2900c170431f-kube-api-access-l424k\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:52.435646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.435638    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:52.435646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.435649    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd477a5-35e1-4ed9-a51a-2900c170431f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:52.571826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:52.571793    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81454f3-adfb-47bd-b59c-20bac4481974" path="/var/lib/kubelet/pods/d81454f3-adfb-47bd-b59c-20bac4481974/volumes"
Nov 03 13:18:53.071873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.071840    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"
Nov 03 13:18:53.072077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.071840    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl" event={"ID":"bcd477a5-35e1-4ed9-a51a-2900c170431f","Type":"ContainerDied","Data":"bc7fea63bae94d996f4203ff612be0e68f98d5c089e5b461c05e3e4813144a1c"}
Nov 03 13:18:53.072077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.071979    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7fea63bae94d996f4203ff612be0e68f98d5c089e5b461c05e3e4813144a1c"
Nov 03 13:18:53.462854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.462824    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 13:18:53.546355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546316    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546356    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546379    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546440    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546481    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546511    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546531    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546582    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.546631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546609    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token\") pod \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\" (UID: \"5dae7f1a-c57a-4707-8f78-03a0464be8f6\") "
Nov 03 13:18:53.547072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546703    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:53.547072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546850    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:53.547072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546704    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:53.547072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.547063    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.547245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.546991    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:53.547245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.547083    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:53.547245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.547090    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.549456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.549413    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:18:53.549456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.549435    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:53.549604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.549505    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:53.549604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.549525    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l" pod "5dae7f1a-c57a-4707-8f78-03a0464be8f6" (UID: "5dae7f1a-c57a-4707-8f78-03a0464be8f6"). InnerVolumeSpecName "kube-api-access-2k89l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:53.625907 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.625806    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"]
Nov 03 13:18:53.640925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.640881    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"]
Nov 03 13:18:53.647703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647673    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.647703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647701    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.647925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647713    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.647925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647723    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.647925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647732    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dae7f1a-c57a-4707-8f78-03a0464be8f6-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.647925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647742    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k89l\" (UniqueName: \"kubernetes.io/projected/5dae7f1a-c57a-4707-8f78-03a0464be8f6-kube-api-access-2k89l\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.647925 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.647771    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dae7f1a-c57a-4707-8f78-03a0464be8f6-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:53.650217 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:53.650187    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"]
Nov 03 13:18:54.079850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.079816    4546 generic.go:356] "Generic (PLEG): container finished" podID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerID="4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372" exitCode=0
Nov 03 13:18:54.080029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.079865    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" event={"ID":"5dae7f1a-c57a-4707-8f78-03a0464be8f6","Type":"ContainerDied","Data":"4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372"}
Nov 03 13:18:54.080029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.079915    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh" event={"ID":"5dae7f1a-c57a-4707-8f78-03a0464be8f6","Type":"ContainerDied","Data":"dcc43d80f4338d49a70013642c3b19c979b8d4e46bacb137c670c9d78513a6c0"}
Nov 03 13:18:54.080029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.079869    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"
Nov 03 13:18:54.080029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.079945    4546 scope.go:122] "RemoveContainer" containerID="4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372"
Nov 03 13:18:54.090592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.090574    4546 scope.go:122] "RemoveContainer" containerID="b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"
Nov 03 13:18:54.100204 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.100184    4546 scope.go:122] "RemoveContainer" containerID="4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372"
Nov 03 13:18:54.100525 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:54.100500    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372\": container with ID starting with 4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372 not found: ID does not exist" containerID="4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372"
Nov 03 13:18:54.100571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.100536    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372"} err="failed to get container status \"4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372\": rpc error: code = NotFound desc = could not find container \"4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372\": container with ID starting with 4bd222caa18fe4e19ecc1165b6f1a7a6bae4c860e2e16e92a41531c39e1f1372 not found: ID does not exist"
Nov 03 13:18:54.100571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.100554    4546 scope.go:122] "RemoveContainer" containerID="b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"
Nov 03 13:18:54.100852 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:18:54.100833    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31\": container with ID starting with b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31 not found: ID does not exist" containerID="b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"
Nov 03 13:18:54.100906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.100859    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31"} err="failed to get container status \"b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31\": rpc error: code = NotFound desc = could not find container \"b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31\": container with ID starting with b0f03d7043c82fc0121bd94246f480446424efed4d9e9f8a91eeaad9e3a00c31 not found: ID does not exist"
Nov 03 13:18:54.237513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.237474    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"]
Nov 03 13:18:54.240035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.240005    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-743d9b-openshift-default-58646685f-22jkh"]
Nov 03 13:18:54.572993 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:54.572958    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" path="/var/lib/kubelet/pods/5dae7f1a-c57a-4707-8f78-03a0464be8f6/volumes"
Nov 03 13:18:56.480934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:56.480885    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:18:58.277120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:58.277075    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"]
Nov 03 13:18:58.277644 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:18:58.277373    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy" containerID="cri-o://24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177" gracePeriod=30
Nov 03 13:19:00.968859 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:00.968819    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 13:19:01.024499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.024404    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume\") pod \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") "
Nov 03 13:19:01.024499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.024453    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7\" (UniqueName: \"kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7\") pod \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\" (UID: \"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956\") "
Nov 03 13:19:01.024918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.024885    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume" pod "ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" (UID: "ab4dc0d7-09a8-453a-94c0-e2e5a02a5956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:01.024992 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.024977    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:01.026996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.026972    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7" pod "ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" (UID: "ab4dc0d7-09a8-453a-94c0-e2e5a02a5956"). InnerVolumeSpecName "kube-api-access-tjzh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:01.125995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.125960    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjzh7\" (UniqueName: \"kubernetes.io/projected/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956-kube-api-access-tjzh7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:01.132002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.131968    4546 generic.go:356] "Generic (PLEG): container finished" podID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerID="6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273" exitCode=0
Nov 03 13:19:01.132183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.132020    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"
Nov 03 13:19:01.132183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.132057    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" event={"ID":"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956","Type":"ContainerDied","Data":"6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273"}
Nov 03 13:19:01.132183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.132115    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b" event={"ID":"ab4dc0d7-09a8-453a-94c0-e2e5a02a5956","Type":"ContainerDied","Data":"dd0187b89f8489cd32973beff68ffd73217dda13b303ecbb384520e8f4a962e6"}
Nov 03 13:19:01.132183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.132137    4546 scope.go:122] "RemoveContainer" containerID="6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273"
Nov 03 13:19:01.142476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.142451    4546 scope.go:122] "RemoveContainer" containerID="6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273"
Nov 03 13:19:01.142813 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:01.142787    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273\": container with ID starting with 6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273 not found: ID does not exist" containerID="6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273"
Nov 03 13:19:01.142915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.142822    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273"} err="failed to get container status \"6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273\": rpc error: code = NotFound desc = could not find container \"6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273\": container with ID starting with 6bfecacda9301ff224bc7c587f5765bfb09572c30dc7cc1f204d40d716239273 not found: ID does not exist"
Nov 03 13:19:01.286176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.286077    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"]
Nov 03 13:19:01.290063 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:01.290032    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-743d9b-648df9956f-tf58b"]
Nov 03 13:19:02.022714 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.022680    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"]
Nov 03 13:19:02.050162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050125    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"]
Nov 03 13:19:02.050534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050522    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bcd477a5-35e1-4ed9-a51a-2900c170431f" containerName="revoke-keys"
Nov 03 13:19:02.050582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050536    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd477a5-35e1-4ed9-a51a-2900c170431f" containerName="revoke-keys"
Nov 03 13:19:02.050582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050554    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy"
Nov 03 13:19:02.050582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050559    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy"
Nov 03 13:19:02.050582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050568    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy"
Nov 03 13:19:02.050582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050572    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy"
Nov 03 13:19:02.050782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050591    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerName="payload-processing"
Nov 03 13:19:02.050782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050596    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerName="payload-processing"
Nov 03 13:19:02.050782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050651    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd477a5-35e1-4ed9-a51a-2900c170431f" containerName="revoke-keys"
Nov 03 13:19:02.050782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050662    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy"
Nov 03 13:19:02.050782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050672    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dae7f1a-c57a-4707-8f78-03a0464be8f6" containerName="istio-proxy"
Nov 03 13:19:02.050782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.050681    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" containerName="payload-processing"
Nov 03 13:19:02.055317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.055294    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.068627 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.068597    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"]
Nov 03 13:19:02.072062 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.072040    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8da864c-ptqc6"]
Nov 03 13:19:02.112198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.112158    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:19:02.112198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.112185    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:19:02.112198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.112188    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:19:02.115076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.115046    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"]
Nov 03 13:19:02.132614 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.132548    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"]
Nov 03 13:19:02.135264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.135233    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.135474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.135349    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwq46\" (UniqueName: \"kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.135474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.135425    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.202656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.202626    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 13:19:02.236305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.236271    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.236305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.236310    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.236533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.236368    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwq46\" (UniqueName: \"kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.236938 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.236906    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.238826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.238801    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.289810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.289710    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwq46\" (UniqueName: \"kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46\") pod \"maas-api-revoke-sub-d16e4d03-qw7qw\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.369100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.369068    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:02.572275 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.572177    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d359c3-db4d-4592-a4fd-a24453efdb8a" path="/var/lib/kubelet/pods/20d359c3-db4d-4592-a4fd-a24453efdb8a/volumes"
Nov 03 13:19:02.572563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.572547    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4dc0d7-09a8-453a-94c0-e2e5a02a5956" path="/var/lib/kubelet/pods/ab4dc0d7-09a8-453a-94c0-e2e5a02a5956/volumes"
Nov 03 13:19:02.637031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.637001    4546 scope.go:122] "RemoveContainer" containerID="0e2d06bf0a64fca1d2cb63593167d09fb133ce395b334c4064ab176c1420fa5a"
Nov 03 13:19:02.738681 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:02.738647    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"]
Nov 03 13:19:02.739220 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:19:02.739191    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb51a646_a7a1_47b0_9ac3_d6bc639e3924.slice/crio-65e7e03e5f717b49dd8f000ed9eb6145ad33a68ff86f0bdd6406a1e70fca7320 WatchSource:0}: Error finding container 65e7e03e5f717b49dd8f000ed9eb6145ad33a68ff86f0bdd6406a1e70fca7320: Status 404 returned error can't find the container with id 65e7e03e5f717b49dd8f000ed9eb6145ad33a68ff86f0bdd6406a1e70fca7320
Nov 03 13:19:03.170946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.170881    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw" event={"ID":"eb51a646-a7a1-47b0-9ac3-d6bc639e3924","Type":"ContainerStarted","Data":"43a78ae1e514a7ea70c3427c0f7af6fa46d9dde0750a2530e74ad3e4d988dbc5"}
Nov 03 13:19:03.171562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.170980    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw" event={"ID":"eb51a646-a7a1-47b0-9ac3-d6bc639e3924","Type":"ContainerStarted","Data":"65e7e03e5f717b49dd8f000ed9eb6145ad33a68ff86f0bdd6406a1e70fca7320"}
Nov 03 13:19:03.278713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.278655    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw" podStartSLOduration=1.2786389919999999 podStartE2EDuration="1.278638992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:03.275808679 +0000 UTC m=+5284.847392615" watchObservedRunningTime="2026-11-03 13:19:03.278638992 +0000 UTC m=+5284.850222904"
Nov 03 13:19:03.541129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.541061    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 13:19:03.649617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649570    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.649861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649635    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.649861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649742    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.649861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649809    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649870    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649898    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649925    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649986    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.649986    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650003    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:03.650043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650019    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo\") pod \"d5890d46-715d-45cb-8121-d744306340eb\" (UID: \"d5890d46-715d-45cb-8121-d744306340eb\") "
Nov 03 13:19:03.650593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650047    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:03.650593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650357    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:03.650593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650409    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.650593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650429    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.650593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.650442    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.651167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.651141    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:03.653133 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.653109    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:03.653530 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.653508    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:03.654082 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.654049    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "kube-api-access-2nfmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:03.654297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.654266    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo" pod "d5890d46-715d-45cb-8121-d744306340eb" (UID: "d5890d46-715d-45cb-8121-d744306340eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:19:03.751582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.751548    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5890d46-715d-45cb-8121-d744306340eb-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.751582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.751579    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfmh\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-kube-api-access-2nfmh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.751582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.751591    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5890d46-715d-45cb-8121-d744306340eb-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.751868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.751601    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.751868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.751610    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5890d46-715d-45cb-8121-d744306340eb-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:03.751868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:03.751618    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5890d46-715d-45cb-8121-d744306340eb-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:04.041902 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.041868    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"]
Nov 03 13:19:04.044449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.044418    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2679896-gz4cv"]
Nov 03 13:19:04.178676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.178645    4546 generic.go:356] "Generic (PLEG): container finished" podID="d5890d46-715d-45cb-8121-d744306340eb" containerID="24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177" exitCode=0
Nov 03 13:19:04.179190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.178731    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"
Nov 03 13:19:04.179190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.178734    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" event={"ID":"d5890d46-715d-45cb-8121-d744306340eb","Type":"ContainerDied","Data":"24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177"}
Nov 03 13:19:04.179190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.178833    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk" event={"ID":"d5890d46-715d-45cb-8121-d744306340eb","Type":"ContainerDied","Data":"b3684089ed31c5cc1fab41eef0bd88b4015e886c184a1ef08cdea60ae917d474"}
Nov 03 13:19:04.179190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.178869    4546 scope.go:122] "RemoveContainer" containerID="24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177"
Nov 03 13:19:04.180888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.180724    4546 generic.go:356] "Generic (PLEG): container finished" podID="eb51a646-a7a1-47b0-9ac3-d6bc639e3924" containerID="43a78ae1e514a7ea70c3427c0f7af6fa46d9dde0750a2530e74ad3e4d988dbc5" exitCode=0
Nov 03 13:19:04.180888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.180787    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw" event={"ID":"eb51a646-a7a1-47b0-9ac3-d6bc639e3924","Type":"ContainerDied","Data":"43a78ae1e514a7ea70c3427c0f7af6fa46d9dde0750a2530e74ad3e4d988dbc5"}
Nov 03 13:19:04.189551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.189536    4546 scope.go:122] "RemoveContainer" containerID="24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177"
Nov 03 13:19:04.189858 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:04.189837    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177\": container with ID starting with 24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177 not found: ID does not exist" containerID="24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177"
Nov 03 13:19:04.189909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.189868    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177"} err="failed to get container status \"24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177\": rpc error: code = NotFound desc = could not find container \"24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177\": container with ID starting with 24ec39b86e65cddb6903aba5678f15b7e2e499ec1efef3834d9a075c1727a177 not found: ID does not exist"
Nov 03 13:19:04.437555 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.437468    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"]
Nov 03 13:19:04.483111 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.483075    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-zbbbk"]
Nov 03 13:19:04.571741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.571708    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5890d46-715d-45cb-8121-d744306340eb" path="/var/lib/kubelet/pods/d5890d46-715d-45cb-8121-d744306340eb/volumes"
Nov 03 13:19:04.572160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:04.572145    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df167a5d-7243-428b-94c8-d08a59a28993" path="/var/lib/kubelet/pods/df167a5d-7243-428b-94c8-d08a59a28993/volumes"
Nov 03 13:19:05.324720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.324695    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:05.367051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.367019    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca\") pod \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") "
Nov 03 13:19:05.367246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.367078    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca\") pod \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") "
Nov 03 13:19:05.367246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.367195    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46\" (UniqueName: \"kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46\") pod \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\" (UID: \"eb51a646-a7a1-47b0-9ac3-d6bc639e3924\") "
Nov 03 13:19:05.367486 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.367455    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca" pod "eb51a646-a7a1-47b0-9ac3-d6bc639e3924" (UID: "eb51a646-a7a1-47b0-9ac3-d6bc639e3924"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:05.367600 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.367560    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:05.369388 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.369360    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca" pod "eb51a646-a7a1-47b0-9ac3-d6bc639e3924" (UID: "eb51a646-a7a1-47b0-9ac3-d6bc639e3924"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:05.369478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.369384    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46" pod "eb51a646-a7a1-47b0-9ac3-d6bc639e3924" (UID: "eb51a646-a7a1-47b0-9ac3-d6bc639e3924"). InnerVolumeSpecName "kube-api-access-qwq46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:05.468267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.468229    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:05.468267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:05.468264    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwq46\" (UniqueName: \"kubernetes.io/projected/eb51a646-a7a1-47b0-9ac3-d6bc639e3924-kube-api-access-qwq46\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:06.194692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:06.194650    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"
Nov 03 13:19:06.194692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:06.194678    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw" event={"ID":"eb51a646-a7a1-47b0-9ac3-d6bc639e3924","Type":"ContainerDied","Data":"65e7e03e5f717b49dd8f000ed9eb6145ad33a68ff86f0bdd6406a1e70fca7320"}
Nov 03 13:19:06.194945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:06.194725    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e7e03e5f717b49dd8f000ed9eb6145ad33a68ff86f0bdd6406a1e70fca7320"
Nov 03 13:19:07.023336 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:07.023292    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" containerName="payload-processing" containerID="cri-o://30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85" gracePeriod=25
Nov 03 13:19:07.133403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:07.133335    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerName="payload-pre-processing" containerID="cri-o://f59b22a269c85edb2b1b6b4cec0a9d032ffa317ac230c767cf81b87148ae7d19" gracePeriod=25
Nov 03 13:19:07.650728 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:07.650689    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:19:08.647215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.647168    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" containerID="cri-o://51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf" gracePeriod=45
Nov 03 13:19:08.661615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.661579    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" containerID="cri-o://66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6" gracePeriod=45
Nov 03 13:19:08.670507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.670472    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main" containerID="cri-o://daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d" gracePeriod=45
Nov 03 13:19:08.675194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.675162    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused"
Nov 03 13:19:08.744199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.744150    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": dial tcp 10.133.0.99:8000: connect: connection refused"
Nov 03 13:19:08.940320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.940292    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 13:19:08.943503 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.943479    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 13:19:08.973630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:08.973598    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 13:19:09.001045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001015    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc\" (UniqueName: \"kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc\") pod \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") "
Nov 03 13:19:09.001212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001060    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs\" (UniqueName: \"kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs\") pod \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") "
Nov 03 13:19:09.001212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001084    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home\") pod \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") "
Nov 03 13:19:09.001212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001135    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm\") pod \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") "
Nov 03 13:19:09.001212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001180    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home\") pod \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001215    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir\") pod \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001244    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir\") pod \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001268    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs\") pod \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001316    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache\") pod \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\" (UID: \"8e30c5cb-3d9b-49da-af00-2c2ac2253a47\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001345    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6\" (UniqueName: \"kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6\") pod \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001372    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache\") pod \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001419    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm\") pod \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\" (UID: \"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22\") "
Nov 03 13:19:09.001474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001420    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home" pod "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" (UID: "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001486    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home" pod "8e30c5cb-3d9b-49da-af00-2c2ac2253a47" (UID: "8e30c5cb-3d9b-49da-af00-2c2ac2253a47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001628    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache" pod "8e30c5cb-3d9b-49da-af00-2c2ac2253a47" (UID: "8e30c5cb-3d9b-49da-af00-2c2ac2253a47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001770    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir" pod "8e30c5cb-3d9b-49da-af00-2c2ac2253a47" (UID: "8e30c5cb-3d9b-49da-af00-2c2ac2253a47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001883    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001905    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001922    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.001946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001939    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.002514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.001563    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir" pod "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" (UID: "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.002514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.002053    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache" pod "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" (UID: "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.004007 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.003981    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs" pod "8e30c5cb-3d9b-49da-af00-2c2ac2253a47" (UID: "8e30c5cb-3d9b-49da-af00-2c2ac2253a47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:09.004180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.004025    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm" pod "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" (UID: "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.004180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.004109    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs" pod "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" (UID: "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:09.004180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.004136    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc" pod "8e30c5cb-3d9b-49da-af00-2c2ac2253a47" (UID: "8e30c5cb-3d9b-49da-af00-2c2ac2253a47"). InnerVolumeSpecName "kube-api-access-xd4sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:09.004401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.004380    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6" pod "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" (UID: "566a9ae7-d9c9-40fb-8436-5f2fa22ecb22"). InnerVolumeSpecName "kube-api-access-sfhb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:09.004460 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.004444    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm" pod "8e30c5cb-3d9b-49da-af00-2c2ac2253a47" (UID: "8e30c5cb-3d9b-49da-af00-2c2ac2253a47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.102780 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.102725    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home\") pod \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") "
Nov 03 13:19:09.102780 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.102783    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm\") pod \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") "
Nov 03 13:19:09.103044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.102816    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir\") pod \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") "
Nov 03 13:19:09.103044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.102885    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs\") pod \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") "
Nov 03 13:19:09.103044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.102957    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2\" (UniqueName: \"kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2\") pod \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") "
Nov 03 13:19:09.103044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.102988    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache\") pod \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\" (UID: \"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450\") "
Nov 03 13:19:09.103279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103099    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir" pod "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" (UID: "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.103279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103110    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home" pod "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" (UID: "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.103389 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103335    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache" pod "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" (UID: "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.103464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103441    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4sc\" (UniqueName: \"kubernetes.io/projected/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-kube-api-access-xd4sc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103465    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103481    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103495    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103513    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103674 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103539    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e30c5cb-3d9b-49da-af00-2c2ac2253a47-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103674 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103617    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103712    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfhb6\" (UniqueName: \"kubernetes.io/projected/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-kube-api-access-sfhb6\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103725    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103740    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.103773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.103770    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.105152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.105127    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm" pod "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" (UID: "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.105762 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.105724    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs" pod "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" (UID: "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:09.105887 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.105764    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2" pod "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" (UID: "d4e4feb2-507e-44a8-b9e0-5ac70b9a3450"). InnerVolumeSpecName "kube-api-access-fvrg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:09.163087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.162988    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:19:09.204385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.204345    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrg2\" (UniqueName: \"kubernetes.io/projected/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-kube-api-access-fvrg2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.204385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.204378    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.204385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.204391    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.212419 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.212391    4546 generic.go:356] "Generic (PLEG): container finished" podID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerID="66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6" exitCode=0
Nov 03 13:19:09.212589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.212444    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"
Nov 03 13:19:09.212589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.212485    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" event={"ID":"8e30c5cb-3d9b-49da-af00-2c2ac2253a47","Type":"ContainerDied","Data":"66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6"}
Nov 03 13:19:09.212589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.212551    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m" event={"ID":"8e30c5cb-3d9b-49da-af00-2c2ac2253a47","Type":"ContainerDied","Data":"717387b3bdd7ef00e1e7b755f4b014fc0fc4c3a269ae814fc28c12db98985c7c"}
Nov 03 13:19:09.212589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.212580    4546 scope.go:122] "RemoveContainer" containerID="66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6"
Nov 03 13:19:09.214424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.214395    4546 generic.go:356] "Generic (PLEG): container finished" podID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerID="daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d" exitCode=0
Nov 03 13:19:09.214563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.214446    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"
Nov 03 13:19:09.214563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.214491    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" event={"ID":"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450","Type":"ContainerDied","Data":"daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d"}
Nov 03 13:19:09.214563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.214530    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68" event={"ID":"d4e4feb2-507e-44a8-b9e0-5ac70b9a3450","Type":"ContainerDied","Data":"aa639fee78e662ea72e5f147be7e1e7373d6d93b3bc85530d037c2a4b75bfc7f"}
Nov 03 13:19:09.216132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.216110    4546 generic.go:356] "Generic (PLEG): container finished" podID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerID="51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf" exitCode=0
Nov 03 13:19:09.216264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.216218    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" event={"ID":"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22","Type":"ContainerDied","Data":"51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf"}
Nov 03 13:19:09.216264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.216247    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp" event={"ID":"566a9ae7-d9c9-40fb-8436-5f2fa22ecb22","Type":"ContainerDied","Data":"71a2b8a61c37873608e7772c3a034b7b417a7b0c0cc6bb3ce8bb3cfe75bb437e"}
Nov 03 13:19:09.216264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.216222    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"
Nov 03 13:19:09.223233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.223195    4546 scope.go:122] "RemoveContainer" containerID="66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6"
Nov 03 13:19:09.223533 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:09.223490    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6\": container with ID starting with 66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6 not found: ID does not exist" containerID="66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6"
Nov 03 13:19:09.223631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.223529    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6"} err="failed to get container status \"66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6\": rpc error: code = NotFound desc = could not find container \"66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6\": container with ID starting with 66357cd69b723b2ffa402d58962742305748b138d685e7bf2f4f2e078de08ba6 not found: ID does not exist"
Nov 03 13:19:09.223631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.223547    4546 scope.go:122] "RemoveContainer" containerID="daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d"
Nov 03 13:19:09.234053 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.234031    4546 scope.go:122] "RemoveContainer" containerID="daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d"
Nov 03 13:19:09.234382 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:09.234350    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d\": container with ID starting with daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d not found: ID does not exist" containerID="daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d"
Nov 03 13:19:09.234466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.234388    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d"} err="failed to get container status \"daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d\": rpc error: code = NotFound desc = could not find container \"daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d\": container with ID starting with daae2de296433771dbaadf664691372a8269fffb7bfdb0c21dedc9a540a2be8d not found: ID does not exist"
Nov 03 13:19:09.234466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.234406    4546 scope.go:122] "RemoveContainer" containerID="51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf"
Nov 03 13:19:09.243176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.243156    4546 scope.go:122] "RemoveContainer" containerID="51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf"
Nov 03 13:19:09.243480 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:09.243454    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf\": container with ID starting with 51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf not found: ID does not exist" containerID="51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf"
Nov 03 13:19:09.243584 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.243490    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf"} err="failed to get container status \"51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf\": rpc error: code = NotFound desc = could not find container \"51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf\": container with ID starting with 51112fdffd21075cff54950dde4986b6b9de51912b711eb7b170a3cdd64d2ecf not found: ID does not exist"
Nov 03 13:19:09.366804 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.366770    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"]
Nov 03 13:19:09.368980 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.368956    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/simulator-w0-9893f0-kserve-684c984b74-nnl68"]
Nov 03 13:19:09.469418 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.469380    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"]
Nov 03 13:19:09.475282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.475254    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/premium-w0-9893f0-kserve-6779bb7f9-mg72m"]
Nov 03 13:19:09.577633 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.577597    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"]
Nov 03 13:19:09.581498 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:09.581474    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-2-w0-9893f0-kserve-677ff789c7-xfbrp"]
Nov 03 13:19:10.177508 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:10.177473    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:19:10.571731 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:10.571694    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" path="/var/lib/kubelet/pods/566a9ae7-d9c9-40fb-8436-5f2fa22ecb22/volumes"
Nov 03 13:19:10.572158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:10.572144    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" path="/var/lib/kubelet/pods/8e30c5cb-3d9b-49da-af00-2c2ac2253a47/volumes"
Nov 03 13:19:10.572507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:10.572494    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" path="/var/lib/kubelet/pods/d4e4feb2-507e-44a8-b9e0-5ac70b9a3450/volumes"
Nov 03 13:19:14.052332 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:14.052293    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"]
Nov 03 13:19:14.098721 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:14.098679    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"]
Nov 03 13:19:14.100876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:14.100846    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8775045f-mmzbl"]
Nov 03 13:19:14.102762 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:14.102717    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237a96c-52l24"]
Nov 03 13:19:14.572128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:14.572085    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5" path="/var/lib/kubelet/pods/5486f5a0-2107-4ab0-9a5b-2cc662dc9ad5/volumes"
Nov 03 13:19:14.572601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:14.572580    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc98211-1a38-412b-b4aa-258282063efd" path="/var/lib/kubelet/pods/8bc98211-1a38-412b-b4aa-258282063efd/volumes"
Nov 03 13:19:19.162199 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:19.162163    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:19:20.177459 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:20.177412    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:19:25.044955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:25.043407    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"]
Nov 03 13:19:25.087723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:25.087683    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fceff6-qjpw5"]
Nov 03 13:19:26.571739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:26.571693    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8809a730-43d4-4ada-8f92-5acb35d72b13" path="/var/lib/kubelet/pods/8809a730-43d4-4ada-8f92-5acb35d72b13/volumes"
Nov 03 13:19:27.297388 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.297310    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 13:19:27.319601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.319566    4546 generic.go:356] "Generic (PLEG): container finished" podID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerID="f59b22a269c85edb2b1b6b4cec0a9d032ffa317ac230c767cf81b87148ae7d19" exitCode=0
Nov 03 13:19:27.319776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.319632    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" event={"ID":"05da420e-d1f6-446c-8256-0ee3bf572eae","Type":"ContainerDied","Data":"f59b22a269c85edb2b1b6b4cec0a9d032ffa317ac230c767cf81b87148ae7d19"}
Nov 03 13:19:27.321248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.321224    4546 generic.go:356] "Generic (PLEG): container finished" podID="28468bae-eedf-4f25-a758-18a876eed1fc" containerID="30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85" exitCode=0
Nov 03 13:19:27.321362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.321293    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" event={"ID":"28468bae-eedf-4f25-a758-18a876eed1fc","Type":"ContainerDied","Data":"30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85"}
Nov 03 13:19:27.321362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.321342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm" event={"ID":"28468bae-eedf-4f25-a758-18a876eed1fc","Type":"ContainerDied","Data":"7168f955a3b89e9f6ac36c7ca3f7fee5f4aafb8c1ccd4f09a4d0aa49256080da"}
Nov 03 13:19:27.321362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.321353    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"
Nov 03 13:19:27.321495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.321375    4546 scope.go:122] "RemoveContainer" containerID="30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85"
Nov 03 13:19:27.332882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.332853    4546 scope.go:122] "RemoveContainer" containerID="30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85"
Nov 03 13:19:27.333327 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:27.333296    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85\": container with ID starting with 30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85 not found: ID does not exist" containerID="30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85"
Nov 03 13:19:27.333454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.333345    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85"} err="failed to get container status \"30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85\": rpc error: code = NotFound desc = could not find container \"30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85\": container with ID starting with 30c9a454de1bac896fd5c2561acd887807b0f9932d09be47712918f0980cee85 not found: ID does not exist"
Nov 03 13:19:27.373181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.373155    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns\" (UniqueName: \"kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns\") pod \"28468bae-eedf-4f25-a758-18a876eed1fc\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") "
Nov 03 13:19:27.373334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.373249    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume\") pod \"28468bae-eedf-4f25-a758-18a876eed1fc\" (UID: \"28468bae-eedf-4f25-a758-18a876eed1fc\") "
Nov 03 13:19:27.373683 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.373656    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume" pod "28468bae-eedf-4f25-a758-18a876eed1fc" (UID: "28468bae-eedf-4f25-a758-18a876eed1fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:27.375475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.375449    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns" pod "28468bae-eedf-4f25-a758-18a876eed1fc" (UID: "28468bae-eedf-4f25-a758-18a876eed1fc"). InnerVolumeSpecName "kube-api-access-fd4ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:27.379839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.379819    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 13:19:27.474391 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.474355    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt\" (UniqueName: \"kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt\") pod \"05da420e-d1f6-446c-8256-0ee3bf572eae\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") "
Nov 03 13:19:27.474616 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.474413    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume\") pod \"05da420e-d1f6-446c-8256-0ee3bf572eae\" (UID: \"05da420e-d1f6-446c-8256-0ee3bf572eae\") "
Nov 03 13:19:27.474739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.474720    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd4ns\" (UniqueName: \"kubernetes.io/projected/28468bae-eedf-4f25-a758-18a876eed1fc-kube-api-access-fd4ns\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:27.474828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.474770    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28468bae-eedf-4f25-a758-18a876eed1fc-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:27.474866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.474828    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume" pod "05da420e-d1f6-446c-8256-0ee3bf572eae" (UID: "05da420e-d1f6-446c-8256-0ee3bf572eae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:27.476640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.476621    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt" pod "05da420e-d1f6-446c-8256-0ee3bf572eae" (UID: "05da420e-d1f6-446c-8256-0ee3bf572eae"). InnerVolumeSpecName "kube-api-access-x7tlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:27.575785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.575673    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05da420e-d1f6-446c-8256-0ee3bf572eae-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:27.575785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.575706    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tlt\" (UniqueName: \"kubernetes.io/projected/05da420e-d1f6-446c-8256-0ee3bf572eae-kube-api-access-x7tlt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:27.733395 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.733360    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"]
Nov 03 13:19:27.738461 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:27.738425    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9893f0-5bd78446d9-bwhrm"]
Nov 03 13:19:28.329057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.328974    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8" event={"ID":"05da420e-d1f6-446c-8256-0ee3bf572eae","Type":"ContainerDied","Data":"39f8bc1a5c382bb70cbdc682934e9f34c47c61d49a3733cafb6d7e25a3959bf0"}
Nov 03 13:19:28.329057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.329003    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"
Nov 03 13:19:28.329057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.329044    4546 scope.go:122] "RemoveContainer" containerID="f59b22a269c85edb2b1b6b4cec0a9d032ffa317ac230c767cf81b87148ae7d19"
Nov 03 13:19:28.481896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.481860    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"]
Nov 03 13:19:28.528102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.528071    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9893f0-5799d84867-nz5x8"]
Nov 03 13:19:28.571633 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.571601    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" path="/var/lib/kubelet/pods/05da420e-d1f6-446c-8256-0ee3bf572eae/volumes"
Nov 03 13:19:28.572047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:28.572032    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" path="/var/lib/kubelet/pods/28468bae-eedf-4f25-a758-18a876eed1fc/volumes"
Nov 03 13:19:34.586550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:34.586516    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"]
Nov 03 13:19:34.587214 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:34.586703    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy" containerID="cri-o://8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f" gracePeriod=30
Nov 03 13:19:36.567359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:36.567332    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-7749m_b5f43aa3-c9b8-4f13-bfa7-d875cb364992/kuadrant-console-plugin/0.log"
Nov 03 13:19:37.650804 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:37.650770    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:19:38.868313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:38.868274    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cd79d8544-f6nxm_abc3e369-07e6-4332-a68a-58c77461df97/manager/0.log"
Nov 03 13:19:39.844639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.844613    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 13:19:39.992496 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992464    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992568    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992624    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992662    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992712    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992743    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992800    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992847    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992880    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.992957    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data\") pod \"9db91edd-3539-4929-aa6e-64b719feef91\" (UID: \"9db91edd-3539-4929-aa6e-64b719feef91\") "
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993026    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993057    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993261    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993444    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993481    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993500    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993509    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db91edd-3539-4929-aa6e-64b719feef91-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:39.993554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.993456    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:39.995566 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.995536    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:19:39.995690 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.995580    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:39.995690 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.995646    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:39.996465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:39.996440    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh" pod "9db91edd-3539-4929-aa6e-64b719feef91" (UID: "9db91edd-3539-4929-aa6e-64b719feef91"). InnerVolumeSpecName "kube-api-access-pjflh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:40.094249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.094216    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:40.094249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.094249    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/9db91edd-3539-4929-aa6e-64b719feef91-kube-api-access-pjflh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:40.094249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.094260    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db91edd-3539-4929-aa6e-64b719feef91-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:40.094524 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.094269    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:40.094524 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.094277    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db91edd-3539-4929-aa6e-64b719feef91-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:40.392837 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.392730    4546 generic.go:356] "Generic (PLEG): container finished" podID="9db91edd-3539-4929-aa6e-64b719feef91" containerID="8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f" exitCode=0
Nov 03 13:19:40.392837 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.392800    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"
Nov 03 13:19:40.392837 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.392814    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" event={"ID":"9db91edd-3539-4929-aa6e-64b719feef91","Type":"ContainerDied","Data":"8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f"}
Nov 03 13:19:40.393161 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.392873    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px" event={"ID":"9db91edd-3539-4929-aa6e-64b719feef91","Type":"ContainerDied","Data":"ba7bad8b76591c3d6e51d237ad38d06ae1352dc40e6bee80e1fc38c7dd1f39f4"}
Nov 03 13:19:40.393161 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.392897    4546 scope.go:122] "RemoveContainer" containerID="8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f"
Nov 03 13:19:40.403002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.402979    4546 scope.go:122] "RemoveContainer" containerID="8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f"
Nov 03 13:19:40.403295 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:19:40.403275    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f\": container with ID starting with 8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f not found: ID does not exist" containerID="8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f"
Nov 03 13:19:40.403364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.403309    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f"} err="failed to get container status \"8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f\": rpc error: code = NotFound desc = could not find container \"8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f\": container with ID starting with 8a544beb95fcbd377b835fc801895c139b2305c7781a1f32003d13b06b09605f not found: ID does not exist"
Nov 03 13:19:40.545896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.545863    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"]
Nov 03 13:19:40.592370 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:40.592331    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-9s2px"]
Nov 03 13:19:42.571915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:42.571883    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db91edd-3539-4929-aa6e-64b719feef91" path="/var/lib/kubelet/pods/9db91edd-3539-4929-aa6e-64b719feef91/volumes"
Nov 03 13:19:46.387414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387376    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"]
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387781    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387793    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387803    4546 cpu_manager.go:414] "containerMap: removing container" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387808    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387819    4546 cpu_manager.go:414] "containerMap: removing container" podUID="eb51a646-a7a1-47b0-9ac3-d6bc639e3924" containerName="revoke-keys"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387823    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb51a646-a7a1-47b0-9ac3-d6bc639e3924" containerName="revoke-keys"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387831    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387836    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387842    4546 cpu_manager.go:414] "containerMap: removing container" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerName="payload-pre-processing"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387847    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerName="payload-pre-processing"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387860    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387865    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387872    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy"
Nov 03 13:19:46.387870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387877    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387888    4546 cpu_manager.go:414] "containerMap: removing container" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" containerName="payload-processing"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387893    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" containerName="payload-processing"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387944    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db91edd-3539-4929-aa6e-64b719feef91" containerName="istio-proxy"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387952    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="566a9ae7-d9c9-40fb-8436-5f2fa22ecb22" containerName="main"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387959    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb51a646-a7a1-47b0-9ac3-d6bc639e3924" containerName="revoke-keys"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387964    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e30c5cb-3d9b-49da-af00-2c2ac2253a47" containerName="main"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387969    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="05da420e-d1f6-446c-8256-0ee3bf572eae" containerName="payload-pre-processing"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387975    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e4feb2-507e-44a8-b9e0-5ac70b9a3450" containerName="main"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387982    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5890d46-715d-45cb-8121-d744306340eb" containerName="istio-proxy"
Nov 03 13:19:46.388347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.387988    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="28468bae-eedf-4f25-a758-18a876eed1fc" containerName="payload-processing"
Nov 03 13:19:46.391296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.391275    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.440033 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.439993    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-67188486-openshift-default-dockercfg-q7hd2\""
Nov 03 13:19:46.449396 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449356    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449396 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449404    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449614 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449540    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwcl\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449614 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449567    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449623    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449656    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449723    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449785    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.449860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.449835    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.467072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.467040    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 13:19:46.489161 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.489125    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"]
Nov 03 13:19:46.550309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550277    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550314    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550353    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwcl\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550371    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550397    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550417    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550562    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550613    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550642    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.550989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.550857    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.551153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.551032    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.551153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.551061    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.551153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.551120    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.551153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.551143    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.552866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.552840    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.553266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.553243    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.604297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.604259    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.604472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.604318    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwcl\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl\") pod \"e2e-ait-67188486-openshift-default-fbc968bb9-b9c66\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") " pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:46.705200 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:46.705114    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:47.121655 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.121629    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"]
Nov 03 13:19:47.123668 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:19:47.123619    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50040f7e_de21_40a5_8660_7ec3164114a3.slice/crio-05230a2c21f5211ef29d6c97dc9eb5b1878ba2f6078d4a279daa1b2b85498373 WatchSource:0}: Error finding container 05230a2c21f5211ef29d6c97dc9eb5b1878ba2f6078d4a279daa1b2b85498373: Status 404 returned error can't find the container with id 05230a2c21f5211ef29d6c97dc9eb5b1878ba2f6078d4a279daa1b2b85498373
Nov 03 13:19:47.126146 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.126113    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:19:47.126248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.126181    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:19:47.126248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.126209    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:19:47.433338 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.433246    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" event={"ID":"50040f7e-de21-40a5-8660-7ec3164114a3","Type":"ContainerStarted","Data":"e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"}
Nov 03 13:19:47.433338 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.433309    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" event={"ID":"50040f7e-de21-40a5-8660-7ec3164114a3","Type":"ContainerStarted","Data":"05230a2c21f5211ef29d6c97dc9eb5b1878ba2f6078d4a279daa1b2b85498373"}
Nov 03 13:19:47.543469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.543416    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podStartSLOduration=1.543400558 podStartE2EDuration="1.543400558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:47.542259661 +0000 UTC m=+5329.113843573" watchObservedRunningTime="2026-11-03 13:19:47.543400558 +0000 UTC m=+5329.114984506"
Nov 03 13:19:47.705633 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:47.705541    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:48.711115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:48.711088    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:49.443353 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:49.443319    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:49.444376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:49.444349    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:19:53.150159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.150117    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"]
Nov 03 13:19:53.154463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.154422    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.202636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.202597    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.202937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.202664    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt\") pod \"payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.203232 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.203209    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67188486\""
Nov 03 13:19:53.249259 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.249220    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67188486-dockercfg-tpqzn\""
Nov 03 13:19:53.251469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.251446    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"]
Nov 03 13:19:53.303850 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.303810    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt\") pod \"payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.304060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.303945    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.304705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.304685    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.360388 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.360350    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt\") pod \"payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.470249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.470217    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:53.833107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:53.833082    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"]
Nov 03 13:19:53.835526 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:19:53.835495    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode845f358_4055_45d1_baf5_d179dd647cda.slice/crio-5e6511ef56473105f56b818c571590f4c0b70ab3ca79cbd1b4fc419739a3bc76 WatchSource:0}: Error finding container 5e6511ef56473105f56b818c571590f4c0b70ab3ca79cbd1b4fc419739a3bc76: Status 404 returned error can't find the container with id 5e6511ef56473105f56b818c571590f4c0b70ab3ca79cbd1b4fc419739a3bc76
Nov 03 13:19:54.469175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:54.469132    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp" event={"ID":"e845f358-4055-45d1-baf5-d179dd647cda","Type":"ContainerStarted","Data":"80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b"}
Nov 03 13:19:54.469699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:54.469202    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp" event={"ID":"e845f358-4055-45d1-baf5-d179dd647cda","Type":"ContainerStarted","Data":"5e6511ef56473105f56b818c571590f4c0b70ab3ca79cbd1b4fc419739a3bc76"}
Nov 03 13:19:54.469699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:54.469329    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:19:54.579270 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:19:54.579215    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp" podStartSLOduration=1.579202497 podStartE2EDuration="1.579202497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:54.577193275 +0000 UTC m=+5336.148777188" watchObservedRunningTime="2026-11-03 13:19:54.579202497 +0000 UTC m=+5336.150786409"
Nov 03 13:20:02.686770 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:02.686710    4546 scope.go:122] "RemoveContainer" containerID="539a357ea4e2fe76208d9cfa92b0038e10e9ce238624acc5e632abc1ecfe73cc"
Nov 03 13:20:02.695981 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:02.695957    4546 scope.go:122] "RemoveContainer" containerID="83a52842e1a7b3d5c3c7f6a09c02e3bf64696537c770b4edd09dede615572c51"
Nov 03 13:20:02.704231 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:02.704210    4546 scope.go:122] "RemoveContainer" containerID="2a38dc57d9bd33399b67a86e818dd06ec554f213993fb721f9265e81a217ce6d"
Nov 03 13:20:05.477610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:05.477570    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:20:07.657018 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:20:07.656940    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:20:12.579984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:12.579953    4546 generic.go:356] "Generic (PLEG): container finished" podID="50040f7e-de21-40a5-8660-7ec3164114a3" containerID="e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd" exitCode=0
Nov 03 13:20:12.580432 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:12.580021    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" event={"ID":"50040f7e-de21-40a5-8660-7ec3164114a3","Type":"ContainerDied","Data":"e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"}
Nov 03 13:20:12.580432 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:12.580380    4546 scope.go:122] "RemoveContainer" containerID="e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"
Nov 03 13:20:12.581639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:12.581610    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:20:12.581718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:12.581682    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:20:12.581718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:12.581714    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:20:13.586784 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:13.586731    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" event={"ID":"50040f7e-de21-40a5-8660-7ec3164114a3","Type":"ContainerStarted","Data":"d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731"}
Nov 03 13:20:13.706247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:13.706217    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:20:13.706431 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:13.706356    4546 patch_prober.go:29] interesting pod/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused" start-of-body=
Nov 03 13:20:13.706431 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:13.706415    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused"
Nov 03 13:20:14.705841 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:14.705797    4546 patch_prober.go:29] interesting pod/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused" start-of-body=
Nov 03 13:20:14.706364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:14.705874    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused"
Nov 03 13:20:15.706318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:15.706284    4546 patch_prober.go:29] interesting pod/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused" start-of-body=
Nov 03 13:20:15.706809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:15.706339    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused"
Nov 03 13:20:16.706180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:16.706146    4546 patch_prober.go:29] interesting pod/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused" start-of-body=
Nov 03 13:20:16.706376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:16.706201    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused"
Nov 03 13:20:17.706079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:17.706040    4546 patch_prober.go:29] interesting pod/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused" start-of-body=
Nov 03 13:20:17.706280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:17.706096    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.110:15021/healthz/ready\": dial tcp 10.133.1.110:15021: connect: connection refused"
Nov 03 13:20:18.744649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:18.744610    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:20:18.745128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:18.744890    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:20:18.745772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:18.745726    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:20:26.037730 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.037694    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"]
Nov 03 13:20:26.041529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.041507    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.095273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.095237    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"]
Nov 03 13:20:26.139279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.139240    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:20:26.139279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.139244    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:20:26.139516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.139306    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:20:26.139516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.139364    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:20:26.147022 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.146994    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.147107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.147053    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nljr\" (UniqueName: \"kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.147107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.147074    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.248180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.248136    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.248381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.248202    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nljr\" (UniqueName: \"kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.248381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.248223    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.248900 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.248875    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.250840 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.250816    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.301952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.301873    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nljr\" (UniqueName: \"kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr\") pod \"maas-api-revoke-keys-e2e-ait-67188486-v6ksl\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.355194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.355138    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:26.722176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:26.722147    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"]
Nov 03 13:20:26.723938 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:20:26.723909    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c305bcd_c1e9_486a_8e62_32356cc8a234.slice/crio-4bc63f16f9020c77c76c30b03a73883f361c5d8fcb48ee42e5f0eb2ea1d4a7f0 WatchSource:0}: Error finding container 4bc63f16f9020c77c76c30b03a73883f361c5d8fcb48ee42e5f0eb2ea1d4a7f0: Status 404 returned error can't find the container with id 4bc63f16f9020c77c76c30b03a73883f361c5d8fcb48ee42e5f0eb2ea1d4a7f0
Nov 03 13:20:27.655541 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:27.655505    4546 generic.go:356] "Generic (PLEG): container finished" podID="8c305bcd-c1e9-486a-8e62-32356cc8a234" containerID="5b481511bbac487ae90b412e6eedd7d9f1e723043fdb33d0d6dd2c8b4a2e2891" exitCode=0
Nov 03 13:20:27.655969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:27.655560    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl" event={"ID":"8c305bcd-c1e9-486a-8e62-32356cc8a234","Type":"ContainerDied","Data":"5b481511bbac487ae90b412e6eedd7d9f1e723043fdb33d0d6dd2c8b4a2e2891"}
Nov 03 13:20:27.655969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:27.655605    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl" event={"ID":"8c305bcd-c1e9-486a-8e62-32356cc8a234","Type":"ContainerStarted","Data":"4bc63f16f9020c77c76c30b03a73883f361c5d8fcb48ee42e5f0eb2ea1d4a7f0"}
Nov 03 13:20:28.797765 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.797722    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:28.872404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.872367    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca\") pod \"8c305bcd-c1e9-486a-8e62-32356cc8a234\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") "
Nov 03 13:20:28.872589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.872435    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr\" (UniqueName: \"kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr\") pod \"8c305bcd-c1e9-486a-8e62-32356cc8a234\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") "
Nov 03 13:20:28.872589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.872494    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca\") pod \"8c305bcd-c1e9-486a-8e62-32356cc8a234\" (UID: \"8c305bcd-c1e9-486a-8e62-32356cc8a234\") "
Nov 03 13:20:28.872945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.872909    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca" pod "8c305bcd-c1e9-486a-8e62-32356cc8a234" (UID: "8c305bcd-c1e9-486a-8e62-32356cc8a234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:28.874689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.874658    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca" pod "8c305bcd-c1e9-486a-8e62-32356cc8a234" (UID: "8c305bcd-c1e9-486a-8e62-32356cc8a234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:28.874795 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.874775    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr" pod "8c305bcd-c1e9-486a-8e62-32356cc8a234" (UID: "8c305bcd-c1e9-486a-8e62-32356cc8a234"). InnerVolumeSpecName "kube-api-access-7nljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:28.973706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.973668    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:28.973706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.973705    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nljr\" (UniqueName: \"kubernetes.io/projected/8c305bcd-c1e9-486a-8e62-32356cc8a234-kube-api-access-7nljr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:28.973954 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:28.973715    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c305bcd-c1e9-486a-8e62-32356cc8a234-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:29.666648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:29.666615    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"
Nov 03 13:20:29.666873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:29.666614    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl" event={"ID":"8c305bcd-c1e9-486a-8e62-32356cc8a234","Type":"ContainerDied","Data":"4bc63f16f9020c77c76c30b03a73883f361c5d8fcb48ee42e5f0eb2ea1d4a7f0"}
Nov 03 13:20:29.666873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:29.666743    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bc63f16f9020c77c76c30b03a73883f361c5d8fcb48ee42e5f0eb2ea1d4a7f0"
Nov 03 13:20:32.043619 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:32.043572    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"]
Nov 03 13:20:32.089447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:32.089417    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921daae4-qzffn"]
Nov 03 13:20:32.572291 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:32.572258    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51747e1-6638-441f-9db0-04e57347cf7c" path="/var/lib/kubelet/pods/f51747e1-6638-441f-9db0-04e57347cf7c/volumes"
Nov 03 13:20:37.636167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:37.636132    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"]
Nov 03 13:20:37.651668 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:20:37.651634    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:20:41.955532 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:41.955489    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"]
Nov 03 13:20:42.001667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:42.001628    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67188486-v6ksl"]
Nov 03 13:20:42.572773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:42.572712    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c305bcd-c1e9-486a-8e62-32356cc8a234" path="/var/lib/kubelet/pods/8c305bcd-c1e9-486a-8e62-32356cc8a234/volumes"
Nov 03 13:20:42.637150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:42.637100    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp" podUID="e845f358-4055-45d1-baf5-d179dd647cda" containerName="payload-pre-processing" containerID="cri-o://80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b" gracePeriod=25
Nov 03 13:20:42.885059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:42.885031    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:20:43.012527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.012489    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt\") pod \"e845f358-4055-45d1-baf5-d179dd647cda\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") "
Nov 03 13:20:43.013068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.012591    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume\") pod \"e845f358-4055-45d1-baf5-d179dd647cda\" (UID: \"e845f358-4055-45d1-baf5-d179dd647cda\") "
Nov 03 13:20:43.013139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.013076    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume" pod "e845f358-4055-45d1-baf5-d179dd647cda" (UID: "e845f358-4055-45d1-baf5-d179dd647cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:43.015219 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.015195    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt" pod "e845f358-4055-45d1-baf5-d179dd647cda" (UID: "e845f358-4055-45d1-baf5-d179dd647cda"). InnerVolumeSpecName "kube-api-access-jsjvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:43.049705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.049661    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"]
Nov 03 13:20:43.052228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.052201    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553d398e-k2hd8"]
Nov 03 13:20:43.113314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.113223    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/e845f358-4055-45d1-baf5-d179dd647cda-kube-api-access-jsjvt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:43.113314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.113255    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e845f358-4055-45d1-baf5-d179dd647cda-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:43.735586 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.735551    4546 generic.go:356] "Generic (PLEG): container finished" podID="e845f358-4055-45d1-baf5-d179dd647cda" containerID="80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b" exitCode=0
Nov 03 13:20:43.735810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.735599    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"
Nov 03 13:20:43.735810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.735640    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp" event={"ID":"e845f358-4055-45d1-baf5-d179dd647cda","Type":"ContainerDied","Data":"80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b"}
Nov 03 13:20:43.735810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.735697    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp" event={"ID":"e845f358-4055-45d1-baf5-d179dd647cda","Type":"ContainerDied","Data":"5e6511ef56473105f56b818c571590f4c0b70ab3ca79cbd1b4fc419739a3bc76"}
Nov 03 13:20:43.735810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.735721    4546 scope.go:122] "RemoveContainer" containerID="80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b"
Nov 03 13:20:43.747488 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.745934    4546 scope.go:122] "RemoveContainer" containerID="80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b"
Nov 03 13:20:43.747488 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:20:43.747436    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b\": container with ID starting with 80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b not found: ID does not exist" containerID="80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b"
Nov 03 13:20:43.747488 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.747469    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b"} err="failed to get container status \"80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b\": rpc error: code = NotFound desc = could not find container \"80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b\": container with ID starting with 80c6856b4d4d852ee059527b65c6631d9bc1566b27615cacffe818b5d5fddc9b not found: ID does not exist"
Nov 03 13:20:43.889068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.889025    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"]
Nov 03 13:20:43.895436 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:43.895401    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67188486-df87f9db7-dzffp"]
Nov 03 13:20:44.571716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:44.571678    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f23854-8d8e-466b-af64-a1b69bd2aa4e" path="/var/lib/kubelet/pods/71f23854-8d8e-466b-af64-a1b69bd2aa4e/volumes"
Nov 03 13:20:44.572180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:44.572085    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e845f358-4055-45d1-baf5-d179dd647cda" path="/var/lib/kubelet/pods/e845f358-4055-45d1-baf5-d179dd647cda/volumes"
Nov 03 13:20:47.198483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:47.198446    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"]
Nov 03 13:20:47.198939 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:47.198668    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" containerID="cri-o://d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731" gracePeriod=30
Nov 03 13:20:48.748222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:48.748180    4546 patch_prober.go:29] interesting pod/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:20:48.748715 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:48.748239    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:20:52.441384 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.441348    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:20:52.501638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501610    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.501838 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501664    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.501838 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501704    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.501838 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501827    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.502001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501858    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.502001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501903    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.502001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501953    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.502001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.501974    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:20:52.502233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502020    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.502233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502046    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert\") pod \"50040f7e-de21-40a5-8660-7ec3164114a3\" (UID: \"50040f7e-de21-40a5-8660-7ec3164114a3\") "
Nov 03 13:20:52.502233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502139    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:20:52.502407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502338    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:20:52.502407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502397    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.502504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502407    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:20:52.502504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502421    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.502504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.502461    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:52.504346 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.504322    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:20:52.504473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.504370    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "kube-api-access-8pwcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:52.504551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.504532    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:20:52.504612 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.504549    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token" pod "50040f7e-de21-40a5-8660-7ec3164114a3" (UID: "50040f7e-de21-40a5-8660-7ec3164114a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:52.604296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604220    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50040f7e-de21-40a5-8660-7ec3164114a3-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.604563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604256    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.604686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604576    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.604686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604595    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.604686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604608    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50040f7e-de21-40a5-8660-7ec3164114a3-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.604686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604621    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwcl\" (UniqueName: \"kubernetes.io/projected/50040f7e-de21-40a5-8660-7ec3164114a3-kube-api-access-8pwcl\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.604686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.604637    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50040f7e-de21-40a5-8660-7ec3164114a3-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.782365 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.782278    4546 generic.go:356] "Generic (PLEG): container finished" podID="50040f7e-de21-40a5-8660-7ec3164114a3" containerID="d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731" exitCode=0
Nov 03 13:20:52.782365 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.782334    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"
Nov 03 13:20:52.782576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.782370    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" event={"ID":"50040f7e-de21-40a5-8660-7ec3164114a3","Type":"ContainerDied","Data":"d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731"}
Nov 03 13:20:52.782576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.782426    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66" event={"ID":"50040f7e-de21-40a5-8660-7ec3164114a3","Type":"ContainerDied","Data":"05230a2c21f5211ef29d6c97dc9eb5b1878ba2f6078d4a279daa1b2b85498373"}
Nov 03 13:20:52.782576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.782491    4546 scope.go:122] "RemoveContainer" containerID="d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731"
Nov 03 13:20:52.791709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.791675    4546 scope.go:122] "RemoveContainer" containerID="e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"
Nov 03 13:20:52.799992 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.799970    4546 scope.go:122] "RemoveContainer" containerID="d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731"
Nov 03 13:20:52.800273 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:20:52.800256    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731\": container with ID starting with d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731 not found: ID does not exist" containerID="d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731"
Nov 03 13:20:52.800325 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.800282    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731"} err="failed to get container status \"d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731\": rpc error: code = NotFound desc = could not find container \"d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731\": container with ID starting with d89e95032ae5a2f6104db41b0e288b7b44d0d21d92f7f6fef358002c1815b731 not found: ID does not exist"
Nov 03 13:20:52.800325 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.800299    4546 scope.go:122] "RemoveContainer" containerID="e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"
Nov 03 13:20:52.800512 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:20:52.800495    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd\": container with ID starting with e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd not found: ID does not exist" containerID="e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"
Nov 03 13:20:52.800553 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.800515    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd"} err="failed to get container status \"e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd\": rpc error: code = NotFound desc = could not find container \"e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd\": container with ID starting with e9078f1ba856cb9af84e1fcb91dbf93df1e1a786aca588641f58e5eff87628bd not found: ID does not exist"
Nov 03 13:20:52.936542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.936501    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"]
Nov 03 13:20:52.941436 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:52.941407    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-67188486-openshift-default-fbc968bb9-b9c66"]
Nov 03 13:20:53.291031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.290993    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-f5g5d"]
Nov 03 13:20:53.291357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291344    4546 cpu_manager.go:414] "containerMap: removing container" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy"
Nov 03 13:20:53.291404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291359    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy"
Nov 03 13:20:53.291404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291373    4546 cpu_manager.go:414] "containerMap: removing container" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy"
Nov 03 13:20:53.291404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291378    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291406    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8c305bcd-c1e9-486a-8e62-32356cc8a234" containerName="revoke-keys"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291411    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c305bcd-c1e9-486a-8e62-32356cc8a234" containerName="revoke-keys"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291419    4546 cpu_manager.go:414] "containerMap: removing container" podUID="e845f358-4055-45d1-baf5-d179dd647cda" containerName="payload-pre-processing"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291424    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e845f358-4055-45d1-baf5-d179dd647cda" containerName="payload-pre-processing"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291480    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291489    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" containerName="istio-proxy"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291499    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c305bcd-c1e9-486a-8e62-32356cc8a234" containerName="revoke-keys"
Nov 03 13:20:53.291513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.291505    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="e845f358-4055-45d1-baf5-d179dd647cda" containerName="payload-pre-processing"
Nov 03 13:20:53.296058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.296035    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.311120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.311091    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95kz7\" (UniqueName: \"kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.311260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.311151    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.311260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.311196    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.344811 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.344742    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 13:20:53.345002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.344840    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 13:20:53.345002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.344845    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 13:20:53.390819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.390789    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-htd5n\""
Nov 03 13:20:53.392809 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.392787    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-f5g5d"]
Nov 03 13:20:53.412194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.412158    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kz7\" (UniqueName: \"kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.412378 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.412212    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.412378 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.412268    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.415005 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.414981    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.415158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.415137    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.465237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.465203    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kz7\" (UniqueName: \"kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7\") pod \"maas-controller-78c58c9bfd-f5g5d\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") " pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.608927 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.608827    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:53.972593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.972399    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-f5g5d"]
Nov 03 13:20:53.974973 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:20:53.974944    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d130a04_3428_4b04_8bc6_6cec4147029c.slice/crio-2d76d4596b7d4714c2a3969707f8303ace5989bfdaab0ead058ec6d1bd9822e6 WatchSource:0}: Error finding container 2d76d4596b7d4714c2a3969707f8303ace5989bfdaab0ead058ec6d1bd9822e6: Status 404 returned error can't find the container with id 2d76d4596b7d4714c2a3969707f8303ace5989bfdaab0ead058ec6d1bd9822e6
Nov 03 13:20:53.976406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:53.976383    4546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:20:54.044027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:54.043996    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"]
Nov 03 13:20:54.046601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:54.046575    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f25ca0-62v7p"]
Nov 03 13:20:54.573249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:54.573216    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e1b2d2-3a29-4294-9cdb-e6bf27154d1d" path="/var/lib/kubelet/pods/03e1b2d2-3a29-4294-9cdb-e6bf27154d1d/volumes"
Nov 03 13:20:54.573783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:54.573635    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50040f7e-de21-40a5-8660-7ec3164114a3" path="/var/lib/kubelet/pods/50040f7e-de21-40a5-8660-7ec3164114a3/volumes"
Nov 03 13:20:54.795279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:54.795246    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" event={"ID":"6d130a04-3428-4b04-8bc6-6cec4147029c","Type":"ContainerStarted","Data":"2d76d4596b7d4714c2a3969707f8303ace5989bfdaab0ead058ec6d1bd9822e6"}
Nov 03 13:20:59.085698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:59.085659    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 13:20:59.824024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:59.823983    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" event={"ID":"6d130a04-3428-4b04-8bc6-6cec4147029c","Type":"ContainerStarted","Data":"5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97"}
Nov 03 13:20:59.824234 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:59.824086    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:20:59.931349 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:20:59.931292    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" podStartSLOduration=1.870360206 podStartE2EDuration="6.931278173s" totalImagesPullingTime="5.060917967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:59.930667137 +0000 UTC m=+5401.502251048" watchObservedRunningTime="2026-11-03 13:20:59.931278173 +0000 UTC m=+5401.502862111"
Nov 03 13:21:02.762434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:02.762402    4546 scope.go:122] "RemoveContainer" containerID="65c83ddebff2980e7e53f499287ef891185abe909a97422b813671f8146fe84a"
Nov 03 13:21:02.771309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:02.771283    4546 scope.go:122] "RemoveContainer" containerID="4d2dc609ad8231d6f933e66eb78a3b7fac4a24e21e17d194322081503e79a2ad"
Nov 03 13:21:02.779858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:02.779736    4546 scope.go:122] "RemoveContainer" containerID="2888dd93e980a8b0fdc47c8fef795653250f4fe6e293ae29b1be1b7c5a2d9abe"
Nov 03 13:21:07.651712 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:21:07.651672    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:21:10.043351 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:10.043311    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"]
Nov 03 13:21:10.090230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:10.090184    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a151f42b-zfl2d"]
Nov 03 13:21:10.571833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:10.571802    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db33051e-e188-4d16-9b7c-04b3821a972f" path="/var/lib/kubelet/pods/db33051e-e188-4d16-9b7c-04b3821a972f/volumes"
Nov 03 13:21:10.832863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:10.832776    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:21:17.005801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.005767    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"]
Nov 03 13:21:17.009441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.009423    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.058284 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.058245    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 13:21:17.104293 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.104261    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg57t\""
Nov 03 13:21:17.108437 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.108407    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"]
Nov 03 13:21:17.134473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134437    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134475    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134554    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64h6t\" (UniqueName: \"kubernetes.io/projected/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-kube-api-access-64h6t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134575    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134606    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134665    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134711    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134742    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.134979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.134790    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235263    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235299    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235321    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235374    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235395    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235434    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64h6t\" (UniqueName: \"kubernetes.io/projected/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-kube-api-access-64h6t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235455    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235496    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235685    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235718    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235826    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.235923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.235848    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.236160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.236023    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.236160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.236117    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.238039 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.238017    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.238162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.238092    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.290842 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.290765    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64h6t\" (UniqueName: \"kubernetes.io/projected/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-kube-api-access-64h6t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.291035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.291015    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d962e47d-d4cc-4fa6-b010-1ebf687b3c9e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hsknc\" (UID: \"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.323067 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.323025    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:17.736175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.736151    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"]
Nov 03 13:21:17.738999 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:21:17.738966    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd962e47d_d4cc_4fa6_b010_1ebf687b3c9e.slice/crio-70ee229c52af0e55451aecaa2146a2bd3a0757295ecd66b7c36cd1fcab3b8bab WatchSource:0}: Error finding container 70ee229c52af0e55451aecaa2146a2bd3a0757295ecd66b7c36cd1fcab3b8bab: Status 404 returned error can't find the container with id 70ee229c52af0e55451aecaa2146a2bd3a0757295ecd66b7c36cd1fcab3b8bab
Nov 03 13:21:17.741544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.741503    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:21:17.741637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.741589    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:21:17.741637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.741617    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:21:17.912996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.912957    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" event={"ID":"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e","Type":"ContainerStarted","Data":"eb9b7ecd9e444d52cf02cf5264360bf7d265a95395eb1b42396fba7dfe49b1ff"}
Nov 03 13:21:17.913183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:17.913014    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" event={"ID":"d962e47d-d4cc-4fa6-b010-1ebf687b3c9e","Type":"ContainerStarted","Data":"70ee229c52af0e55451aecaa2146a2bd3a0757295ecd66b7c36cd1fcab3b8bab"}
Nov 03 13:21:18.025039 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.024905    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podStartSLOduration=2.024886704 podStartE2EDuration="2.024886704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:18.022302194 +0000 UTC m=+5419.593886126" watchObservedRunningTime="2026-11-03 13:21:18.024886704 +0000 UTC m=+5419.596470618"
Nov 03 13:21:18.261326 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.261276    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"]
Nov 03 13:21:18.265646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.265622    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.320828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.316329    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:21:18.320828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.316400    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:21:18.320828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.316412    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:21:18.320828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.316552    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:21:18.320828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.319338    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"]
Nov 03 13:21:18.323881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.323800    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:18.326264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.326230    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:18.326443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.326302    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:18.348663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.348625    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.348892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.348676    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2k7h\" (UniqueName: \"kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.348892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.348871    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.449696 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.449647    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.449933 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.449723    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.449933 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.449758    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2k7h\" (UniqueName: \"kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.450406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.450372    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.452318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.452292    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.503112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.503069    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2k7h\" (UniqueName: \"kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h\") pod \"maas-api-revoke-sub-85237e45-kvbnl\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.580834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.580724    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:18.971562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:18.971505    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"]
Nov 03 13:21:18.973863 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:21:18.973833    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb982ccaa_6710_4265_930a_9504e6954946.slice/crio-7ff2cfd5085f04bb44719f4fcad6717ce847c4692d30f8bc2fb0b2c51191e4d1 WatchSource:0}: Error finding container 7ff2cfd5085f04bb44719f4fcad6717ce847c4692d30f8bc2fb0b2c51191e4d1: Status 404 returned error can't find the container with id 7ff2cfd5085f04bb44719f4fcad6717ce847c4692d30f8bc2fb0b2c51191e4d1
Nov 03 13:21:19.327773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:19.327208    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:19.327773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:19.327273    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:19.941737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:19.941704    4546 generic.go:356] "Generic (PLEG): container finished" podID="b982ccaa-6710-4265-930a-9504e6954946" containerID="e62b74d4259544531fb16f12b2d3312a67ef1e5a90eab48ab7b094c7842346fb" exitCode=0
Nov 03 13:21:19.941737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:19.941796    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl" event={"ID":"b982ccaa-6710-4265-930a-9504e6954946","Type":"ContainerDied","Data":"e62b74d4259544531fb16f12b2d3312a67ef1e5a90eab48ab7b094c7842346fb"}
Nov 03 13:21:19.942094 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:19.941858    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl" event={"ID":"b982ccaa-6710-4265-930a-9504e6954946","Type":"ContainerStarted","Data":"7ff2cfd5085f04bb44719f4fcad6717ce847c4692d30f8bc2fb0b2c51191e4d1"}
Nov 03 13:21:20.327776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:20.327485    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:20.327776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:20.327549    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:21.099076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.099050    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:21.175323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.175291    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca\") pod \"b982ccaa-6710-4265-930a-9504e6954946\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") "
Nov 03 13:21:21.175511 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.175352    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca\") pod \"b982ccaa-6710-4265-930a-9504e6954946\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") "
Nov 03 13:21:21.175511 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.175467    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h\" (UniqueName: \"kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h\") pod \"b982ccaa-6710-4265-930a-9504e6954946\" (UID: \"b982ccaa-6710-4265-930a-9504e6954946\") "
Nov 03 13:21:21.175722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.175699    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca" pod "b982ccaa-6710-4265-930a-9504e6954946" (UID: "b982ccaa-6710-4265-930a-9504e6954946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:21.175876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.175860    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b982ccaa-6710-4265-930a-9504e6954946-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:21.177622 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.177601    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca" pod "b982ccaa-6710-4265-930a-9504e6954946" (UID: "b982ccaa-6710-4265-930a-9504e6954946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:21.177729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.177707    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h" pod "b982ccaa-6710-4265-930a-9504e6954946" (UID: "b982ccaa-6710-4265-930a-9504e6954946"). InnerVolumeSpecName "kube-api-access-h2k7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:21.277295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.277261    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2k7h\" (UniqueName: \"kubernetes.io/projected/b982ccaa-6710-4265-930a-9504e6954946-kube-api-access-h2k7h\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:21.277295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.277293    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b982ccaa-6710-4265-930a-9504e6954946-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:21.331802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.329232    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:21.331802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.329298    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:21.952626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.952595    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"
Nov 03 13:21:21.952626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.952610    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl" event={"ID":"b982ccaa-6710-4265-930a-9504e6954946","Type":"ContainerDied","Data":"7ff2cfd5085f04bb44719f4fcad6717ce847c4692d30f8bc2fb0b2c51191e4d1"}
Nov 03 13:21:21.952896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:21.952653    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff2cfd5085f04bb44719f4fcad6717ce847c4692d30f8bc2fb0b2c51191e4d1"
Nov 03 13:21:22.328233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:22.328188    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:22.328563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:22.328531    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:23.327188 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:23.327144    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:23.327896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:23.327857    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:24.323832 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:24.323785    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:24.324036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:24.323857    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:25.323714 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:25.323675    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body=
Nov 03 13:21:25.324180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:25.323735    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused"
Nov 03 13:21:27.365009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:27.364962    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:21:27.365478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:27.365050    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:28.367464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:28.367429    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:21:28.367899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:28.367490    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:28.370378 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:28.370355    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:28.370658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:28.370638    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:28.371414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:28.371398    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:21:30.709212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:30.709179    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-f5g5d"]
Nov 03 13:21:30.709648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:30.709388    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" podUID="6d130a04-3428-4b04-8bc6-6cec4147029c" containerName="manager" containerID="cri-o://5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97" gracePeriod=30
Nov 03 13:21:30.830622 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:30.830587    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" podUID="6d130a04-3428-4b04-8bc6-6cec4147029c" containerName="manager" probeResult="failure" output="Get \"http://10.133.1.113:8081/readyz\": dial tcp 10.133.1.113:8081: connect: connection refused"
Nov 03 13:21:30.967738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:30.967716    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:21:31.002046 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.002013    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d130a04-3428-4b04-8bc6-6cec4147029c" containerID="5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97" exitCode=0
Nov 03 13:21:31.002222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.002053    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d"
Nov 03 13:21:31.002222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.002090    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" event={"ID":"6d130a04-3428-4b04-8bc6-6cec4147029c","Type":"ContainerDied","Data":"5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97"}
Nov 03 13:21:31.002222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.002146    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-f5g5d" event={"ID":"6d130a04-3428-4b04-8bc6-6cec4147029c","Type":"ContainerDied","Data":"2d76d4596b7d4714c2a3969707f8303ace5989bfdaab0ead058ec6d1bd9822e6"}
Nov 03 13:21:31.002222 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.002169    4546 scope.go:122] "RemoveContainer" containerID="5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97"
Nov 03 13:21:31.010975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.010956    4546 scope.go:122] "RemoveContainer" containerID="5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97"
Nov 03 13:21:31.011274 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:21:31.011255    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97\": container with ID starting with 5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97 not found: ID does not exist" containerID="5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97"
Nov 03 13:21:31.011351 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.011283    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97"} err="failed to get container status \"5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97\": rpc error: code = NotFound desc = could not find container \"5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97\": container with ID starting with 5fffb494a2baaa119f0ebf447cfdde225178fae47974a7a22aed0318c0009c97 not found: ID does not exist"
Nov 03 13:21:31.076025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.075994    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert\") pod \"6d130a04-3428-4b04-8bc6-6cec4147029c\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") "
Nov 03 13:21:31.076233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.076100    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs\") pod \"6d130a04-3428-4b04-8bc6-6cec4147029c\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") "
Nov 03 13:21:31.076233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.076174    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7\" (UniqueName: \"kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7\") pod \"6d130a04-3428-4b04-8bc6-6cec4147029c\" (UID: \"6d130a04-3428-4b04-8bc6-6cec4147029c\") "
Nov 03 13:21:31.079478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.079416    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs" pod "6d130a04-3428-4b04-8bc6-6cec4147029c" (UID: "6d130a04-3428-4b04-8bc6-6cec4147029c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:31.079629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.079550    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert" pod "6d130a04-3428-4b04-8bc6-6cec4147029c" (UID: "6d130a04-3428-4b04-8bc6-6cec4147029c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:31.079906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.079780    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7" pod "6d130a04-3428-4b04-8bc6-6cec4147029c" (UID: "6d130a04-3428-4b04-8bc6-6cec4147029c"). InnerVolumeSpecName "kube-api-access-95kz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:31.177184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.177154    4546 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-webhook-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:31.177184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.177179    4546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d130a04-3428-4b04-8bc6-6cec4147029c-metrics-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:31.177184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.177191    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95kz7\" (UniqueName: \"kubernetes.io/projected/6d130a04-3428-4b04-8bc6-6cec4147029c-kube-api-access-95kz7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:31.415587 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.415548    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-f5g5d"]
Nov 03 13:21:31.420864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:31.420835    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-f5g5d"]
Nov 03 13:21:32.123099 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123060    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"]
Nov 03 13:21:32.123513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123437    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d130a04-3428-4b04-8bc6-6cec4147029c" containerName="manager"
Nov 03 13:21:32.123513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123449    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d130a04-3428-4b04-8bc6-6cec4147029c" containerName="manager"
Nov 03 13:21:32.123513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123471    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b982ccaa-6710-4265-930a-9504e6954946" containerName="revoke-keys"
Nov 03 13:21:32.123513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123476    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b982ccaa-6710-4265-930a-9504e6954946" containerName="revoke-keys"
Nov 03 13:21:32.123663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123539    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b982ccaa-6710-4265-930a-9504e6954946" containerName="revoke-keys"
Nov 03 13:21:32.123663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.123553    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d130a04-3428-4b04-8bc6-6cec4147029c" containerName="manager"
Nov 03 13:21:32.129175 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.129151    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.221701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.221671    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"]
Nov 03 13:21:32.287074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287036    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287077    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287104    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbr8v\" (UniqueName: \"kubernetes.io/projected/93052df7-c1d3-42d5-8d77-e96e0fd78881-kube-api-access-vbr8v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287217    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287296    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287324    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93052df7-c1d3-42d5-8d77-e96e0fd78881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287380    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287405    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.287544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.287473    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388069    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388158    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388181    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388201    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbr8v\" (UniqueName: \"kubernetes.io/projected/93052df7-c1d3-42d5-8d77-e96e0fd78881-kube-api-access-vbr8v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388219    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388243    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388264    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93052df7-c1d3-42d5-8d77-e96e0fd78881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388488    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388535    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388542    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388666    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388685    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.388965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.388892    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.389100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.389083    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93052df7-c1d3-42d5-8d77-e96e0fd78881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.390819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.390796    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.390908 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.390864    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.441610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.441572    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93052df7-c1d3-42d5-8d77-e96e0fd78881-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.441610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.441572    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbr8v\" (UniqueName: \"kubernetes.io/projected/93052df7-c1d3-42d5-8d77-e96e0fd78881-kube-api-access-vbr8v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xfwjk\" (UID: \"93052df7-c1d3-42d5-8d77-e96e0fd78881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.442419 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.442400    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:32.571330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.571298    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d130a04-3428-4b04-8bc6-6cec4147029c" path="/var/lib/kubelet/pods/6d130a04-3428-4b04-8bc6-6cec4147029c/volumes"
Nov 03 13:21:32.753518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.753482    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-64vqd"]
Nov 03 13:21:32.758990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.758970    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.810490 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.810452    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 13:21:32.813208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.813176    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-64vqd"]
Nov 03 13:21:32.856785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.856703    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-htd5n\""
Nov 03 13:21:32.856785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.856703    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 13:21:32.856785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.856782    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 13:21:32.859101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.859043    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"]
Nov 03 13:21:32.861478 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:21:32.861450    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93052df7_c1d3_42d5_8d77_e96e0fd78881.slice/crio-e516769f4a7a5ecd655de084424a18ffc7d75824b6805465829b6469130e41f1 WatchSource:0}: Error finding container e516769f4a7a5ecd655de084424a18ffc7d75824b6805465829b6469130e41f1: Status 404 returned error can't find the container with id e516769f4a7a5ecd655de084424a18ffc7d75824b6805465829b6469130e41f1
Nov 03 13:21:32.863880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.863849    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:21:32.863952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.863917    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:21:32.863952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.863944    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:21:32.892699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.892661    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp5c\" (UniqueName: \"kubernetes.io/projected/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-kube-api-access-nqp5c\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.892924 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.892707    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-webhook-cert\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.892924 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.892870    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-metrics-certs\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.993635 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.993586    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp5c\" (UniqueName: \"kubernetes.io/projected/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-kube-api-access-nqp5c\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.993852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.993784    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-webhook-cert\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.993852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.993834    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-metrics-certs\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.996410 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.996373    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-webhook-cert\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:32.996554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:32.996437    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-metrics-certs\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:33.014405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.014319    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" event={"ID":"93052df7-c1d3-42d5-8d77-e96e0fd78881","Type":"ContainerStarted","Data":"26041750788b76ad5e3ee1c820911db2aa393df8c381465cb3b7278d82025230"}
Nov 03 13:21:33.014405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.014382    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" event={"ID":"93052df7-c1d3-42d5-8d77-e96e0fd78881","Type":"ContainerStarted","Data":"e516769f4a7a5ecd655de084424a18ffc7d75824b6805465829b6469130e41f1"}
Nov 03 13:21:33.048144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.048118    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp5c\" (UniqueName: \"kubernetes.io/projected/e794e88c-e1c1-4534-9bb5-bbc0a8c40773-kube-api-access-nqp5c\") pod \"maas-controller-78c58c9bfd-64vqd\" (UID: \"e794e88c-e1c1-4534-9bb5-bbc0a8c40773\") " pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:33.072617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.072584    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:33.127878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.127826    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podStartSLOduration=1.127809788 podStartE2EDuration="1.127809788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:33.124547416 +0000 UTC m=+5434.696131355" watchObservedRunningTime="2026-11-03 13:21:33.127809788 +0000 UTC m=+5434.699393700"
Nov 03 13:21:33.442646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.442618    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:33.442805 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.442696    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-64vqd"]
Nov 03 13:21:33.444279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.444255    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:33.444403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:33.444307    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:33.444947 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:21:33.444899    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode794e88c_e1c1_4534_9bb5_bbc0a8c40773.slice/crio-220de00a8083584567693acd6fdcc9b1cb55a321850caa5e23bedfb3f33109e1 WatchSource:0}: Error finding container 220de00a8083584567693acd6fdcc9b1cb55a321850caa5e23bedfb3f33109e1: Status 404 returned error can't find the container with id 220de00a8083584567693acd6fdcc9b1cb55a321850caa5e23bedfb3f33109e1
Nov 03 13:21:34.038141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:34.038098    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-64vqd" event={"ID":"e794e88c-e1c1-4534-9bb5-bbc0a8c40773","Type":"ContainerStarted","Data":"220de00a8083584567693acd6fdcc9b1cb55a321850caa5e23bedfb3f33109e1"}
Nov 03 13:21:34.443620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:34.443584    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:34.444156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:34.443641    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:35.046488 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:35.046451    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-64vqd" event={"ID":"e794e88c-e1c1-4534-9bb5-bbc0a8c40773","Type":"ContainerStarted","Data":"fd46b94da8bd8c056eb1c4a078c3a4ab8754b8da4115f43d60afa218662585e2"}
Nov 03 13:21:35.159079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:35.159019    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78c58c9bfd-64vqd" podStartSLOduration=2.34301199 podStartE2EDuration="3.159002972s" totalImagesPullingTime="815.990982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:35.155558757 +0000 UTC m=+5436.727142688" watchObservedRunningTime="2026-11-03 13:21:35.159002972 +0000 UTC m=+5436.730586885"
Nov 03 13:21:35.443300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:35.443180    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:35.443300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:35.443242    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:36.050974 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:36.050943    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:21:36.443463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:36.443366    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:36.443463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:36.443431    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:37.443265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:37.443225    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:37.443858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:37.443289    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:37.657981 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:21:37.657902    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:21:38.443275 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:38.443235    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:38.443726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:38.443292    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:39.443041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:39.443002    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:39.443259 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:39.443062    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:40.443218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:40.443180    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body=
Nov 03 13:21:40.443727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:40.443246    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused"
Nov 03 13:21:41.447692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:41.447662    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:41.448229 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:41.447950    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:41.448725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:41.448708    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk"
Nov 03 13:21:47.058282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:21:47.058244    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78c58c9bfd-64vqd"
Nov 03 13:22:02.816405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:02.816352    4546 scope.go:122] "RemoveContainer" containerID="a126f00a4c1aa0a6c22791ace84206057048fbab9694e4481664dda54c5a1d06"
Nov 03 13:22:02.829788 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:02.829737    4546 scope.go:122] "RemoveContainer" containerID="921d0cd3f956c38065e3d97f1e09c75befe11c503e8be69095fe413b79bfa494"
Nov 03 13:22:02.843365 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:02.843333    4546 scope.go:122] "RemoveContainer" containerID="dcb0853a89b86e5a245bc65e9637578ff023bb7e20c823864d4d54dc32d67db2"
Nov 03 13:22:07.651363 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:22:07.651307    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:22:08.081085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.081040    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"]
Nov 03 13:22:08.086643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.086620    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.134405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.134361    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.134624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.134432    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.134624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.134538    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnxn\" (UniqueName: \"kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.140482 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.140455    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:08.186448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.186415    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:08.186448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.186410    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:08.186733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.186460    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:22:08.188885 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.188862    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"]
Nov 03 13:22:08.236023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.235988    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.236023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.236025    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnxn\" (UniqueName: \"kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.236247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.236101    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.236793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.236730    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.238681 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.238656    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.292833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.292792    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnxn\" (UniqueName: \"kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn\") pod \"maas-api-revoke-sub-f904bff9-85g24\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.401315 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.401219    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:08.768218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:08.768167    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"]
Nov 03 13:22:08.769559 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:22:08.769529    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd8696a_c834_42ef_a0af_e393963a113f.slice/crio-28707f50b7bb5f78686404b14e8bf8332b4980a736775e64753e7aab7784ba43 WatchSource:0}: Error finding container 28707f50b7bb5f78686404b14e8bf8332b4980a736775e64753e7aab7784ba43: Status 404 returned error can't find the container with id 28707f50b7bb5f78686404b14e8bf8332b4980a736775e64753e7aab7784ba43
Nov 03 13:22:09.229093 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:09.229055    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24" event={"ID":"2dd8696a-c834-42ef-a0af-e393963a113f","Type":"ContainerStarted","Data":"ceb4d408a9dc17064c284a06fdcc3e044662311536aad34bff8acf1243bfec99"}
Nov 03 13:22:09.229286 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:09.229124    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24" event={"ID":"2dd8696a-c834-42ef-a0af-e393963a113f","Type":"ContainerStarted","Data":"28707f50b7bb5f78686404b14e8bf8332b4980a736775e64753e7aab7784ba43"}
Nov 03 13:22:09.337822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:09.337698    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24" podStartSLOduration=1.337682675 podStartE2EDuration="1.337682675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:09.335843953 +0000 UTC m=+5470.907427883" watchObservedRunningTime="2026-11-03 13:22:09.337682675 +0000 UTC m=+5470.909266581"
Nov 03 13:22:10.236524 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:10.236487    4546 generic.go:356] "Generic (PLEG): container finished" podID="2dd8696a-c834-42ef-a0af-e393963a113f" containerID="ceb4d408a9dc17064c284a06fdcc3e044662311536aad34bff8acf1243bfec99" exitCode=0
Nov 03 13:22:10.236958 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:10.236549    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24" event={"ID":"2dd8696a-c834-42ef-a0af-e393963a113f","Type":"ContainerDied","Data":"ceb4d408a9dc17064c284a06fdcc3e044662311536aad34bff8acf1243bfec99"}
Nov 03 13:22:11.383572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.383544    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:11.466460 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.466421    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca\") pod \"2dd8696a-c834-42ef-a0af-e393963a113f\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") "
Nov 03 13:22:11.466660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.466593    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn\" (UniqueName: \"kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn\") pod \"2dd8696a-c834-42ef-a0af-e393963a113f\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") "
Nov 03 13:22:11.466660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.466631    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca\") pod \"2dd8696a-c834-42ef-a0af-e393963a113f\" (UID: \"2dd8696a-c834-42ef-a0af-e393963a113f\") "
Nov 03 13:22:11.467031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.467005    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca" pod "2dd8696a-c834-42ef-a0af-e393963a113f" (UID: "2dd8696a-c834-42ef-a0af-e393963a113f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:11.468736 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.468714    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca" pod "2dd8696a-c834-42ef-a0af-e393963a113f" (UID: "2dd8696a-c834-42ef-a0af-e393963a113f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:11.468860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.468842    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn" pod "2dd8696a-c834-42ef-a0af-e393963a113f" (UID: "2dd8696a-c834-42ef-a0af-e393963a113f"). InnerVolumeSpecName "kube-api-access-wfnxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:11.568347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.568258    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnxn\" (UniqueName: \"kubernetes.io/projected/2dd8696a-c834-42ef-a0af-e393963a113f-kube-api-access-wfnxn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:11.568347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.568289    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:11.568347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:11.568298    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8696a-c834-42ef-a0af-e393963a113f-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:12.249436 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:12.249398    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24" event={"ID":"2dd8696a-c834-42ef-a0af-e393963a113f","Type":"ContainerDied","Data":"28707f50b7bb5f78686404b14e8bf8332b4980a736775e64753e7aab7784ba43"}
Nov 03 13:22:12.249436 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:12.249417    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"
Nov 03 13:22:12.249667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:12.249454    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28707f50b7bb5f78686404b14e8bf8332b4980a736775e64753e7aab7784ba43"
Nov 03 13:22:14.053192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:14.053152    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"]
Nov 03 13:22:14.099134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:14.099089    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed5b204-rwg25"]
Nov 03 13:22:14.572592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:14.572557    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e28be8-9a9e-4ccb-b570-16afdb29590f" path="/var/lib/kubelet/pods/40e28be8-9a9e-4ccb-b570-16afdb29590f/volumes"
Nov 03 13:22:17.462179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.462142    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"]
Nov 03 13:22:17.462615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.462541    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2dd8696a-c834-42ef-a0af-e393963a113f" containerName="revoke-keys"
Nov 03 13:22:17.462615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.462553    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd8696a-c834-42ef-a0af-e393963a113f" containerName="revoke-keys"
Nov 03 13:22:17.462687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.462618    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd8696a-c834-42ef-a0af-e393963a113f" containerName="revoke-keys"
Nov 03 13:22:17.466162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.466142    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.631854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.631806    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e0aa94-18d0-4583-8748-75f6100b6272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.631873    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.631919    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.631998    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkllw\" (UniqueName: \"kubernetes.io/projected/45e0aa94-18d0-4583-8748-75f6100b6272-kube-api-access-hkllw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.632051    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.632092    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632380 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.632133    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632380 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.632193    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e0aa94-18d0-4583-8748-75f6100b6272-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.632380 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.632245    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e0aa94-18d0-4583-8748-75f6100b6272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.665069 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.665032    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"]
Nov 03 13:22:17.733149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733100    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733154    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733196    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e0aa94-18d0-4583-8748-75f6100b6272-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733233    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e0aa94-18d0-4583-8748-75f6100b6272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733278    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e0aa94-18d0-4583-8748-75f6100b6272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733310    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733345    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733387    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkllw\" (UniqueName: \"kubernetes.io/projected/45e0aa94-18d0-4583-8748-75f6100b6272-kube-api-access-hkllw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733425    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733597    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733729    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733808    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.733951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.733838    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.734212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.734187    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e0aa94-18d0-4583-8748-75f6100b6272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.735892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.735868    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e0aa94-18d0-4583-8748-75f6100b6272-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.736003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.735987    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e0aa94-18d0-4583-8748-75f6100b6272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.786557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.786524    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e0aa94-18d0-4583-8748-75f6100b6272-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:17.786723 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:17.786697    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkllw\" (UniqueName: \"kubernetes.io/projected/45e0aa94-18d0-4583-8748-75f6100b6272-kube-api-access-hkllw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zg2zd\" (UID: \"45e0aa94-18d0-4583-8748-75f6100b6272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:18.078724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:18.078634    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:18.498920 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:18.498879    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"]
Nov 03 13:22:18.532650 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:22:18.532621    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e0aa94_18d0_4583_8748_75f6100b6272.slice/crio-61689e7947f7bf23f87de3a15e37856d2b6d980c39619a11f99fbafb7d39a585 WatchSource:0}: Error finding container 61689e7947f7bf23f87de3a15e37856d2b6d980c39619a11f99fbafb7d39a585: Status 404 returned error can't find the container with id 61689e7947f7bf23f87de3a15e37856d2b6d980c39619a11f99fbafb7d39a585
Nov 03 13:22:18.534831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:18.534799    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:22:18.534908 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:18.534870    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:22:18.534945 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:18.534916    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:22:19.284792 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:19.284725    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" event={"ID":"45e0aa94-18d0-4583-8748-75f6100b6272","Type":"ContainerStarted","Data":"63e73580723d2e570ab029a1d8b7775259366d4cda0caa9b81177433ec2f5c9f"}
Nov 03 13:22:19.285058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:19.284812    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" event={"ID":"45e0aa94-18d0-4583-8748-75f6100b6272","Type":"ContainerStarted","Data":"61689e7947f7bf23f87de3a15e37856d2b6d980c39619a11f99fbafb7d39a585"}
Nov 03 13:22:19.440252 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:19.440183    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podStartSLOduration=2.440165816 podStartE2EDuration="2.440165816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:19.439125884 +0000 UTC m=+5481.010709796" watchObservedRunningTime="2026-11-03 13:22:19.440165816 +0000 UTC m=+5481.011749723"
Nov 03 13:22:20.079522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:20.079486    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:20.081100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:20.081075    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:20.081240 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:20.081135    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:21.079692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:21.079646    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:21.080137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:21.079706    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:22.079254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:22.079213    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:22.079429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:22.079271    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:23.080037 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:23.079993    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:23.080478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:23.080066    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:24.079208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:24.079134    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:24.079456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:24.079229    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:25.079609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:25.079572    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:25.080167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:25.079641    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:26.079448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:26.079410    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused" start-of-body=
Nov 03 13:22:26.079677 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:26.079472    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": dial tcp 10.133.1.119:15021: connect: connection refused"
Nov 03 13:22:28.120987 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:28.120945    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:28.121515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:28.121030    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:29.167691 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:29.166862    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:29.167691 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:29.166952    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:29.422039 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:29.421924    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:29.422370 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:29.422330    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:30.168225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:30.168184    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:30.168825 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:30.168256    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:31.048084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.048038    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"]
Nov 03 13:22:31.092997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.092958    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7cdc9a-prjb8"]
Nov 03 13:22:31.209975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.209929    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:31.210449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.210022    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:31.635028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.634996    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:31.635316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.635294    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:31.636000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:31.635983    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd"
Nov 03 13:22:32.572434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:32.572388    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0307d050-38d3-495d-948a-413773d4caed" path="/var/lib/kubelet/pods/0307d050-38d3-495d-948a-413773d4caed/volumes"
Nov 03 13:22:37.653712 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:22:37.653665    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:22:43.358619 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.358578    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"]
Nov 03 13:22:43.362141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.362120    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.411280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.411245    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:43.411280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.411281    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:43.411517 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.411349    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:43.442006 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.441969    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.442006 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.442000    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.442268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.442038    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcpn\" (UniqueName: \"kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.457553 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.457526    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:22:43.459549 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.459528    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"]
Nov 03 13:22:43.543574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.543540    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.543574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.543574    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.543923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.543603    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcpn\" (UniqueName: \"kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.544316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.544289    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.546040 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.546021    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.605630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.605601    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcpn\" (UniqueName: \"kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn\") pod \"maas-api-revoke-sub-f03629db-455zk\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:43.675698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:43.675614    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:44.038311 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:44.038290    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"]
Nov 03 13:22:44.040830 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:22:44.040802    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a13ebf8_7fac_4f2b_89cd_31bea173bfd1.slice/crio-7d3a68c5446f2660282ceda2686966c3dfae3f4ece14ba6a08cc295e03a02622 WatchSource:0}: Error finding container 7d3a68c5446f2660282ceda2686966c3dfae3f4ece14ba6a08cc295e03a02622: Status 404 returned error can't find the container with id 7d3a68c5446f2660282ceda2686966c3dfae3f4ece14ba6a08cc295e03a02622
Nov 03 13:22:44.495337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:44.495300    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk" event={"ID":"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1","Type":"ContainerStarted","Data":"4d6b3d53dbc07ef8864cb72e1845aa52139fcdd41da733be6e33a37371dd9e55"}
Nov 03 13:22:44.495801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:44.495361    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk" event={"ID":"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1","Type":"ContainerStarted","Data":"7d3a68c5446f2660282ceda2686966c3dfae3f4ece14ba6a08cc295e03a02622"}
Nov 03 13:22:44.604490 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:44.604392    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk" podStartSLOduration=1.604377293 podStartE2EDuration="1.604377293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:44.601871598 +0000 UTC m=+5506.173455529" watchObservedRunningTime="2026-11-03 13:22:44.604377293 +0000 UTC m=+5506.175961205"
Nov 03 13:22:45.510108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:45.510068    4546 generic.go:356] "Generic (PLEG): container finished" podID="2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" containerID="4d6b3d53dbc07ef8864cb72e1845aa52139fcdd41da733be6e33a37371dd9e55" exitCode=0
Nov 03 13:22:45.510537 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:45.510160    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk" event={"ID":"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1","Type":"ContainerDied","Data":"4d6b3d53dbc07ef8864cb72e1845aa52139fcdd41da733be6e33a37371dd9e55"}
Nov 03 13:22:46.654834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.654810    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:46.776783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.776678    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca\") pod \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") "
Nov 03 13:22:46.776969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.776797    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca\") pod \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") "
Nov 03 13:22:46.776969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.776857    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn\" (UniqueName: \"kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn\") pod \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\" (UID: \"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1\") "
Nov 03 13:22:46.777122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.777103    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca" pod "2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" (UID: "2a13ebf8-7fac-4f2b-89cd-31bea173bfd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:46.777290 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.777264    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.779178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.779147    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca" pod "2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" (UID: "2a13ebf8-7fac-4f2b-89cd-31bea173bfd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:46.779301 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.779207    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn" pod "2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" (UID: "2a13ebf8-7fac-4f2b-89cd-31bea173bfd1"). InnerVolumeSpecName "kube-api-access-xfcpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:46.878640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.878609    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.878640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:46.878637    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcpn\" (UniqueName: \"kubernetes.io/projected/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1-kube-api-access-xfcpn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:47.523984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:47.523955    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"
Nov 03 13:22:47.524172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:47.523957    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk" event={"ID":"2a13ebf8-7fac-4f2b-89cd-31bea173bfd1","Type":"ContainerDied","Data":"7d3a68c5446f2660282ceda2686966c3dfae3f4ece14ba6a08cc295e03a02622"}
Nov 03 13:22:47.524172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:22:47.524098    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d3a68c5446f2660282ceda2686966c3dfae3f4ece14ba6a08cc295e03a02622"
Nov 03 13:23:02.888764 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:02.888728    4546 scope.go:122] "RemoveContainer" containerID="6cf420871d5ceaaa92a2909649de7d4d89f30be468f3d5f2d417960152897c9f"
Nov 03 13:23:02.980849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:02.980812    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"]
Nov 03 13:23:02.981225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:02.981212    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" containerName="revoke-keys"
Nov 03 13:23:02.981297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:02.981226    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" containerName="revoke-keys"
Nov 03 13:23:02.981336 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:02.981302    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" containerName="revoke-keys"
Nov 03 13:23:02.988107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:02.988084    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.041563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.041522    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"]
Nov 03 13:23:03.082084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.082052    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.082290 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.082099    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.082290 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.082176    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rbr\" (UniqueName: \"kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.085091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.085067    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:23:03.085228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.085118    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:23:03.085228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.085183    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:23:03.085228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.085181    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:23:03.183014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.182925    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.183014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.182983    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.183277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.183019    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rbr\" (UniqueName: \"kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.183710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.183683    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.185498 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.185474    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.236871 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.236835    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rbr\" (UniqueName: \"kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr\") pod \"maas-api-revoke-sub-0165d7d2-2g22w\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.301456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.301416    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:03.671605 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:03.671579    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"]
Nov 03 13:23:03.673990 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:23:03.673962    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d73d8d9_95a5_4181_a77e_c1ee90b573b1.slice/crio-964e5c8081378d4d64d0e0b86910db809ee96b16627d505c4b4c938b2b01fa07 WatchSource:0}: Error finding container 964e5c8081378d4d64d0e0b86910db809ee96b16627d505c4b4c938b2b01fa07: Status 404 returned error can't find the container with id 964e5c8081378d4d64d0e0b86910db809ee96b16627d505c4b4c938b2b01fa07
Nov 03 13:23:04.620817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:04.620784    4546 generic.go:356] "Generic (PLEG): container finished" podID="9d73d8d9-95a5-4181-a77e-c1ee90b573b1" containerID="be516a7c8dcaa8aba2dfc6b3c3762c987f92960bcdb80ee0c0a11ed5aec0a885" exitCode=0
Nov 03 13:23:04.621218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:04.620835    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w" event={"ID":"9d73d8d9-95a5-4181-a77e-c1ee90b573b1","Type":"ContainerDied","Data":"be516a7c8dcaa8aba2dfc6b3c3762c987f92960bcdb80ee0c0a11ed5aec0a885"}
Nov 03 13:23:04.621218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:04.620877    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w" event={"ID":"9d73d8d9-95a5-4181-a77e-c1ee90b573b1","Type":"ContainerStarted","Data":"964e5c8081378d4d64d0e0b86910db809ee96b16627d505c4b4c938b2b01fa07"}
Nov 03 13:23:05.760167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.760140    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:05.911979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.911882    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr\" (UniqueName: \"kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr\") pod \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") "
Nov 03 13:23:05.911979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.911939    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca\") pod \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") "
Nov 03 13:23:05.912193 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.911988    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca\") pod \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\" (UID: \"9d73d8d9-95a5-4181-a77e-c1ee90b573b1\") "
Nov 03 13:23:05.912414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.912391    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca" pod "9d73d8d9-95a5-4181-a77e-c1ee90b573b1" (UID: "9d73d8d9-95a5-4181-a77e-c1ee90b573b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:05.914211 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.914185    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca" pod "9d73d8d9-95a5-4181-a77e-c1ee90b573b1" (UID: "9d73d8d9-95a5-4181-a77e-c1ee90b573b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:05.914337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:05.914320    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr" pod "9d73d8d9-95a5-4181-a77e-c1ee90b573b1" (UID: "9d73d8d9-95a5-4181-a77e-c1ee90b573b1"). InnerVolumeSpecName "kube-api-access-85rbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:06.013165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:06.013133    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rbr\" (UniqueName: \"kubernetes.io/projected/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-kube-api-access-85rbr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:06.013165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:06.013162    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:06.013165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:06.013171    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d73d8d9-95a5-4181-a77e-c1ee90b573b1-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:06.631917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:06.631885    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w" event={"ID":"9d73d8d9-95a5-4181-a77e-c1ee90b573b1","Type":"ContainerDied","Data":"964e5c8081378d4d64d0e0b86910db809ee96b16627d505c4b4c938b2b01fa07"}
Nov 03 13:23:06.631917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:06.631912    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"
Nov 03 13:23:06.632137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:06.631935    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="964e5c8081378d4d64d0e0b86910db809ee96b16627d505c4b4c938b2b01fa07"
Nov 03 13:23:07.650361 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:23:07.650274    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:23:14.747076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.747040    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"]
Nov 03 13:23:14.747597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.747429    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9d73d8d9-95a5-4181-a77e-c1ee90b573b1" containerName="revoke-keys"
Nov 03 13:23:14.747597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.747440    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d73d8d9-95a5-4181-a77e-c1ee90b573b1" containerName="revoke-keys"
Nov 03 13:23:14.747597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.747524    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d73d8d9-95a5-4181-a77e-c1ee90b573b1" containerName="revoke-keys"
Nov 03 13:23:14.750698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.750679    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.800491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.800448    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltxj\" (UniqueName: \"kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.800687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.800525    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.800687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.800613    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.801909 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.801888    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:23:14.804546 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.804525    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:23:14.848419 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.848377    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:23:14.850690 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.850670    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:23:14.852947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.852926    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"]
Nov 03 13:23:14.902015 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.901973    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltxj\" (UniqueName: \"kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.902248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.902057    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.902248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.902105    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.902833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.902807    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.904884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.904857    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:14.955495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:14.955464    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltxj\" (UniqueName: \"kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj\") pod \"maas-api-revoke-sub-6b19338f-skzht\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:15.063247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:15.063156    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:15.428486 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:15.428458    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"]
Nov 03 13:23:15.430911 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:23:15.430874    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9589c8b_b679_4c82_95be_2dfae0f22e43.slice/crio-c25357a9efed4a0467ecbc3215da8e6752f5671fbc441b28cdabd468c8ab6800 WatchSource:0}: Error finding container c25357a9efed4a0467ecbc3215da8e6752f5671fbc441b28cdabd468c8ab6800: Status 404 returned error can't find the container with id c25357a9efed4a0467ecbc3215da8e6752f5671fbc441b28cdabd468c8ab6800
Nov 03 13:23:15.685279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:15.685187    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht" event={"ID":"f9589c8b-b679-4c82-95be-2dfae0f22e43","Type":"ContainerStarted","Data":"c25357a9efed4a0467ecbc3215da8e6752f5671fbc441b28cdabd468c8ab6800"}
Nov 03 13:23:16.691783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:16.691685    4546 generic.go:356] "Generic (PLEG): container finished" podID="f9589c8b-b679-4c82-95be-2dfae0f22e43" containerID="3f1e1a0430a172dc72833b039c417fdbd482ba9bf0dad6a52a2c3dcfbaa7e1e2" exitCode=0
Nov 03 13:23:16.692198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:16.691782    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht" event={"ID":"f9589c8b-b679-4c82-95be-2dfae0f22e43","Type":"ContainerDied","Data":"3f1e1a0430a172dc72833b039c417fdbd482ba9bf0dad6a52a2c3dcfbaa7e1e2"}
Nov 03 13:23:17.050660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.050623    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"]
Nov 03 13:23:17.053858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.053830    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c010f8e-j8sfc"]
Nov 03 13:23:17.837514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.837486    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:17.930481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.930441    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj\" (UniqueName: \"kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj\") pod \"f9589c8b-b679-4c82-95be-2dfae0f22e43\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") "
Nov 03 13:23:17.930671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.930501    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca\") pod \"f9589c8b-b679-4c82-95be-2dfae0f22e43\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") "
Nov 03 13:23:17.930671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.930557    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca\") pod \"f9589c8b-b679-4c82-95be-2dfae0f22e43\" (UID: \"f9589c8b-b679-4c82-95be-2dfae0f22e43\") "
Nov 03 13:23:17.931020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.930995    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca" pod "f9589c8b-b679-4c82-95be-2dfae0f22e43" (UID: "f9589c8b-b679-4c82-95be-2dfae0f22e43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:17.932739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.932709    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca" pod "f9589c8b-b679-4c82-95be-2dfae0f22e43" (UID: "f9589c8b-b679-4c82-95be-2dfae0f22e43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:17.932861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:17.932841    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj" pod "f9589c8b-b679-4c82-95be-2dfae0f22e43" (UID: "f9589c8b-b679-4c82-95be-2dfae0f22e43"). InnerVolumeSpecName "kube-api-access-6ltxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:18.031605 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.031572    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ltxj\" (UniqueName: \"kubernetes.io/projected/f9589c8b-b679-4c82-95be-2dfae0f22e43-kube-api-access-6ltxj\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:18.031605 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.031607    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:18.031881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.031620    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9589c8b-b679-4c82-95be-2dfae0f22e43-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:18.578720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.578679    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72db8557-0ee4-4e3a-b6e9-86415129e9e4" path="/var/lib/kubelet/pods/72db8557-0ee4-4e3a-b6e9-86415129e9e4/volumes"
Nov 03 13:23:18.703631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.703596    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht" event={"ID":"f9589c8b-b679-4c82-95be-2dfae0f22e43","Type":"ContainerDied","Data":"c25357a9efed4a0467ecbc3215da8e6752f5671fbc441b28cdabd468c8ab6800"}
Nov 03 13:23:18.703869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.703642    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"
Nov 03 13:23:18.703869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:18.703646    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25357a9efed4a0467ecbc3215da8e6752f5671fbc441b28cdabd468c8ab6800"
Nov 03 13:23:29.426996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:29.426935    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:23:29.427727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:29.427689    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:30.983181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:30.983142    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"]
Nov 03 13:23:30.983686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:30.983547    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f9589c8b-b679-4c82-95be-2dfae0f22e43" containerName="revoke-keys"
Nov 03 13:23:30.983686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:30.983559    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9589c8b-b679-4c82-95be-2dfae0f22e43" containerName="revoke-keys"
Nov 03 13:23:30.983686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:30.983628    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9589c8b-b679-4c82-95be-2dfae0f22e43" containerName="revoke-keys"
Nov 03 13:23:30.989816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:30.989780    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.087070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.087038    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-b2d3d219-kserve-self-signed-certs\""
Nov 03 13:23:31.089485 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.089458    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"]
Nov 03 13:23:31.103857 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.103820    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.104016 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.103886    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.104016 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.103996    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.104124 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.104050    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwn5t\" (UniqueName: \"kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.104172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.104137    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.104172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.104163    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.205403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205358    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.205654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205432    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.205654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205469    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwn5t\" (UniqueName: \"kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.205654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205498    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.205654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205521    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.205654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205580    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.206016 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205967    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.206016 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.205976    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.206092 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.206041    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.208061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.208039    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.208247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.208227    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.260226 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.260139    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwn5t\" (UniqueName: \"kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t\") pod \"e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.305934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.305894    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:31.586393 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.586311    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"]
Nov 03 13:23:31.590171 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.590148    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.669483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.669450    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-b2d3d219-epp-sa-dockercfg-28jls\""
Nov 03 13:23:31.679070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.679030    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"]
Nov 03 13:23:31.689762 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.689720    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"]
Nov 03 13:23:31.691572 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:23:31.691548    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dab33d1_53ba_47e2_9a71_0855aebbd660.slice/crio-28769a9e7a3c14b2abcce0e5407a0963da274c0e366a1f0db2f256daf09a92b0 WatchSource:0}: Error finding container 28769a9e7a3c14b2abcce0e5407a0963da274c0e366a1f0db2f256daf09a92b0: Status 404 returned error can't find the container with id 28769a9e7a3c14b2abcce0e5407a0963da274c0e366a1f0db2f256daf09a92b0
Nov 03 13:23:31.710279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.710244    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs\") pod \"e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.710421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.710359    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgsg\" (UniqueName: \"kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg\") pod \"e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.801116 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.801083    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh" event={"ID":"9dab33d1-53ba-47e2-9a71-0855aebbd660","Type":"ContainerStarted","Data":"d5082838aa2d6ab180ca1d5467c246c09b7957aa8c2f7878f4b50ce12c715ae7"}
Nov 03 13:23:31.801279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.801123    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh" event={"ID":"9dab33d1-53ba-47e2-9a71-0855aebbd660","Type":"ContainerStarted","Data":"28769a9e7a3c14b2abcce0e5407a0963da274c0e366a1f0db2f256daf09a92b0"}
Nov 03 13:23:31.811568 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.811537    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs\") pod \"e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.811771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.811591    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgsg\" (UniqueName: \"kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg\") pod \"e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.814215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.814183    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs\") pod \"e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.864946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.864861    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgsg\" (UniqueName: \"kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg\") pod \"e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") " pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.905829 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.905788    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:31.909084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:31.909031    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh" podStartSLOduration=1.909013665 podStartE2EDuration="1.909013665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:31.908283653 +0000 UTC m=+5553.479867591" watchObservedRunningTime="2026-11-03 13:23:31.909013665 +0000 UTC m=+5553.480597580"
Nov 03 13:23:32.281896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:32.281866    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"]
Nov 03 13:23:32.282965 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:23:32.282939    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1865c62c_56e9_46ba_92d5_8a2b6d8e867c.slice/crio-0707cfb48ec50878c3624271520c99021b3a36c383ac4bda80f84c0089158af6 WatchSource:0}: Error finding container 0707cfb48ec50878c3624271520c99021b3a36c383ac4bda80f84c0089158af6: Status 404 returned error can't find the container with id 0707cfb48ec50878c3624271520c99021b3a36c383ac4bda80f84c0089158af6
Nov 03 13:23:32.306660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:32.306624    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:32.807180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:32.807141    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" event={"ID":"1865c62c-56e9-46ba-92d5-8a2b6d8e867c","Type":"ContainerStarted","Data":"0707cfb48ec50878c3624271520c99021b3a36c383ac4bda80f84c0089158af6"}
Nov 03 13:23:33.052184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:33.052142    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"]
Nov 03 13:23:33.098666 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:33.098574    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc27b28e-zghpm"]
Nov 03 13:23:34.571935 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:34.571910    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e76445-d308-4ee2-8320-64d038044c41" path="/var/lib/kubelet/pods/07e76445-d308-4ee2-8320-64d038044c41/volumes"
Nov 03 13:23:34.820384 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:34.820290    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" event={"ID":"1865c62c-56e9-46ba-92d5-8a2b6d8e867c","Type":"ContainerStarted","Data":"06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b"}
Nov 03 13:23:34.820586 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:34.820492    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:23:34.928574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:34.928517    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podStartSLOduration=1.654008326 podStartE2EDuration="3.928498874s" totalImagesPullingTime="2.274490548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:34.926661095 +0000 UTC m=+5556.498245025" watchObservedRunningTime="2026-11-03 13:23:34.928498874 +0000 UTC m=+5556.500082787"
Nov 03 13:23:37.699659 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:23:37.699617    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:23:41.306819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:41.306782    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:41.319869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:41.319838    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:41.886701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:41.886673    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:23:53.043109 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:53.043068    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"]
Nov 03 13:23:53.089287 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:53.089254    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50fcaaff-mknxl"]
Nov 03 13:23:54.571606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:23:54.571569    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd477a5-35e1-4ed9-a51a-2900c170431f" path="/var/lib/kubelet/pods/bcd477a5-35e1-4ed9-a51a-2900c170431f/volumes"
Nov 03 13:24:02.909395 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:02.909359    4546 scope.go:122] "RemoveContainer" containerID="de351e6150482abe7de7b70f7c268b0de9cfd88582587abe285b26625cb174d2"
Nov 03 13:24:02.918591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:02.918568    4546 scope.go:122] "RemoveContainer" containerID="2ec06588fb10db0e0cbb034225cd762a9af57c5814f2244802bccb53197e5c42"
Nov 03 13:24:05.828497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:05.828469    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:24:06.043851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:06.043813    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"]
Nov 03 13:24:06.046462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:06.046407    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4d03-qw7qw"]
Nov 03 13:24:06.571889 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:06.571858    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb51a646-a7a1-47b0-9ac3-d6bc639e3924" path="/var/lib/kubelet/pods/eb51a646-a7a1-47b0-9ac3-d6bc639e3924/volumes"
Nov 03 13:24:07.651556 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:24:07.651519    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:24:37.651045 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:24:37.650959    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:24:43.139583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:43.139528    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"]
Nov 03 13:24:43.142151 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:43.142124    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"]
Nov 03 13:24:45.828984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:45.828946    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:24:55.829224 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:55.829180    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:24:58.168706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.168664    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main" containerID="cri-o://06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b" gracePeriod=45
Nov 03 13:24:58.168706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.168660    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh" podUID="9dab33d1-53ba-47e2-9a71-0855aebbd660" containerName="main" containerID="cri-o://d5082838aa2d6ab180ca1d5467c246c09b7957aa8c2f7878f4b50ce12c715ae7" gracePeriod=45
Nov 03 13:24:58.312725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.312697    4546 generic.go:356] "Generic (PLEG): container finished" podID="9dab33d1-53ba-47e2-9a71-0855aebbd660" containerID="d5082838aa2d6ab180ca1d5467c246c09b7957aa8c2f7878f4b50ce12c715ae7" exitCode=0
Nov 03 13:24:58.312940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.312738    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh" event={"ID":"9dab33d1-53ba-47e2-9a71-0855aebbd660","Type":"ContainerDied","Data":"d5082838aa2d6ab180ca1d5467c246c09b7957aa8c2f7878f4b50ce12c715ae7"}
Nov 03 13:24:58.430123 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.430055    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:24:58.466152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466112    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home\") pod \"9dab33d1-53ba-47e2-9a71-0855aebbd660\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") "
Nov 03 13:24:58.466374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466169    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache\") pod \"9dab33d1-53ba-47e2-9a71-0855aebbd660\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") "
Nov 03 13:24:58.466374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466195    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t\" (UniqueName: \"kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t\") pod \"9dab33d1-53ba-47e2-9a71-0855aebbd660\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") "
Nov 03 13:24:58.466374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466226    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs\") pod \"9dab33d1-53ba-47e2-9a71-0855aebbd660\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") "
Nov 03 13:24:58.466374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466262    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm\") pod \"9dab33d1-53ba-47e2-9a71-0855aebbd660\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") "
Nov 03 13:24:58.466374 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466283    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir\") pod \"9dab33d1-53ba-47e2-9a71-0855aebbd660\" (UID: \"9dab33d1-53ba-47e2-9a71-0855aebbd660\") "
Nov 03 13:24:58.466664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466464    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache" pod "9dab33d1-53ba-47e2-9a71-0855aebbd660" (UID: "9dab33d1-53ba-47e2-9a71-0855aebbd660"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:58.466664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466476    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home" pod "9dab33d1-53ba-47e2-9a71-0855aebbd660" (UID: "9dab33d1-53ba-47e2-9a71-0855aebbd660"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:58.466664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466639    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir" pod "9dab33d1-53ba-47e2-9a71-0855aebbd660" (UID: "9dab33d1-53ba-47e2-9a71-0855aebbd660"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:58.466664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466659    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:58.466884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.466680    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:58.468793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.468741    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm" pod "9dab33d1-53ba-47e2-9a71-0855aebbd660" (UID: "9dab33d1-53ba-47e2-9a71-0855aebbd660"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:58.468793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.468786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t" pod "9dab33d1-53ba-47e2-9a71-0855aebbd660" (UID: "9dab33d1-53ba-47e2-9a71-0855aebbd660"). InnerVolumeSpecName "kube-api-access-cwn5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:58.469026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.468813    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs" pod "9dab33d1-53ba-47e2-9a71-0855aebbd660" (UID: "9dab33d1-53ba-47e2-9a71-0855aebbd660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:58.567230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.567193    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwn5t\" (UniqueName: \"kubernetes.io/projected/9dab33d1-53ba-47e2-9a71-0855aebbd660-kube-api-access-cwn5t\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:58.567230 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.567228    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab33d1-53ba-47e2-9a71-0855aebbd660-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:58.567470 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.567241    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:58.567470 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:58.567275    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab33d1-53ba-47e2-9a71-0855aebbd660-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:59.318726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:59.318685    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh" event={"ID":"9dab33d1-53ba-47e2-9a71-0855aebbd660","Type":"ContainerDied","Data":"28769a9e7a3c14b2abcce0e5407a0963da274c0e366a1f0db2f256daf09a92b0"}
Nov 03 13:24:59.318726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:59.318707    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"
Nov 03 13:24:59.318726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:59.318733    4546 scope.go:122] "RemoveContainer" containerID="d5082838aa2d6ab180ca1d5467c246c09b7957aa8c2f7878f4b50ce12c715ae7"
Nov 03 13:24:59.473383 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:59.473344    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"]
Nov 03 13:24:59.519331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:24:59.519295    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-664d6dd7db-m5gfh"]
Nov 03 13:25:00.571710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:00.571673    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dab33d1-53ba-47e2-9a71-0855aebbd660" path="/var/lib/kubelet/pods/9dab33d1-53ba-47e2-9a71-0855aebbd660/volumes"
Nov 03 13:25:02.943716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:02.943683    4546 scope.go:122] "RemoveContainer" containerID="15a4420520c97d5e856b26c468b98288f4fee2c30483d26bb7efb2e55e763f5e"
Nov 03 13:25:02.952434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:02.952409    4546 scope.go:122] "RemoveContainer" containerID="43a78ae1e514a7ea70c3427c0f7af6fa46d9dde0750a2530e74ad3e4d988dbc5"
Nov 03 13:25:02.960786 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:02.960736    4546 scope.go:122] "RemoveContainer" containerID="35e91131ce57228783d80e6a4193fc52e823fa604b8d7aaf7bf3252ca3b652a3"
Nov 03 13:25:02.968681 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:02.968662    4546 scope.go:122] "RemoveContainer" containerID="fdde23b4ae5b55216f1b2afa6f95189dc28c04dc3cbab2c23f05354afbe7fa8b"
Nov 03 13:25:05.844960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:05.844895    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:25:05.845689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:05.845666    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:25:07.651859 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:25:07.651822    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:25:15.828716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:15.828678    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:25:25.828419 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:25.828375    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:25:28.426410 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.426380    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:25:28.495639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.495606    4546 generic.go:356] "Generic (PLEG): container finished" podID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerID="06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b" exitCode=0
Nov 03 13:25:28.495881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.495667    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"
Nov 03 13:25:28.495881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.495698    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" event={"ID":"1865c62c-56e9-46ba-92d5-8a2b6d8e867c","Type":"ContainerDied","Data":"06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b"}
Nov 03 13:25:28.495881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.495824    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s" event={"ID":"1865c62c-56e9-46ba-92d5-8a2b6d8e867c","Type":"ContainerDied","Data":"0707cfb48ec50878c3624271520c99021b3a36c383ac4bda80f84c0089158af6"}
Nov 03 13:25:28.495881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.495856    4546 scope.go:122] "RemoveContainer" containerID="06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b"
Nov 03 13:25:28.509140 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.509112    4546 scope.go:122] "RemoveContainer" containerID="06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b"
Nov 03 13:25:28.509393 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.509350    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs\") pod \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") "
Nov 03 13:25:28.509552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.509438    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg\" (UniqueName: \"kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg\") pod \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\" (UID: \"1865c62c-56e9-46ba-92d5-8a2b6d8e867c\") "
Nov 03 13:25:28.509700 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:25:28.509620    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b\": container with ID starting with 06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b not found: ID does not exist" containerID="06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b"
Nov 03 13:25:28.509700 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.509662    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b"} err="failed to get container status \"06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b\": rpc error: code = NotFound desc = could not find container \"06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b\": container with ID starting with 06229b2abb89302f91704c7356070d26cfa10e6b9e1049d16da21dfbb994987b not found: ID does not exist"
Nov 03 13:25:28.513255 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.513221    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs" pod "1865c62c-56e9-46ba-92d5-8a2b6d8e867c" (UID: "1865c62c-56e9-46ba-92d5-8a2b6d8e867c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:25:28.515098 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.515068    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg" pod "1865c62c-56e9-46ba-92d5-8a2b6d8e867c" (UID: "1865c62c-56e9-46ba-92d5-8a2b6d8e867c"). InnerVolumeSpecName "kube-api-access-tjgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:28.611396 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.611361    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:28.611589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.611416    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgsg\" (UniqueName: \"kubernetes.io/projected/1865c62c-56e9-46ba-92d5-8a2b6d8e867c-kube-api-access-tjgsg\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:28.902620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.902525    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"]
Nov 03 13:25:28.948644 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:28.948608    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-b2d3d219-kserve-router-scheduler-79599ffc7c-qvt6s"]
Nov 03 13:25:30.571549 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:30.571514    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" path="/var/lib/kubelet/pods/1865c62c-56e9-46ba-92d5-8a2b6d8e867c/volumes"
Nov 03 13:25:34.674042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674005    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"]
Nov 03 13:25:34.674456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674375    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main"
Nov 03 13:25:34.674456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674386    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main"
Nov 03 13:25:34.674456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674408    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9dab33d1-53ba-47e2-9a71-0855aebbd660" containerName="main"
Nov 03 13:25:34.674456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674413    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dab33d1-53ba-47e2-9a71-0855aebbd660" containerName="main"
Nov 03 13:25:34.674604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674482    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1865c62c-56e9-46ba-92d5-8a2b6d8e867c" containerName="main"
Nov 03 13:25:34.674604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.674494    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dab33d1-53ba-47e2-9a71-0855aebbd660" containerName="main"
Nov 03 13:25:34.677876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.677855    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.771624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.771582    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.771624 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.771631    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.771888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.771648    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.771888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.771871    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.771982 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.771912    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dv5\" (UniqueName: \"kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.771982 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.771933    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.775956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.775927    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-55973f\"/\"default-dockercfg-ntclb\""
Nov 03 13:25:34.776117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.776017    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-55973f\"/\"simulator-single-main-55973f-kserve-self-signed-certs\""
Nov 03 13:25:34.776241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.776220    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-55973f\"/\"kube-root-ca.crt\""
Nov 03 13:25:34.776308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.776290    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-55973f\"/\"openshift-service-ca.crt\""
Nov 03 13:25:34.777989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.777969    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"]
Nov 03 13:25:34.873508 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.873470    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dv5\" (UniqueName: \"kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.873725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.873518    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.873725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.873576    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.873725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.873603    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.873725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.873632    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.874032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.873801    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.874091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.874039    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.874131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.874098    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.874171 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.874160    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.876289 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.876269    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.876663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.876643    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.927254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.927184    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dv5\" (UniqueName: \"kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5\") pod \"simulator-single-main-55973f-kserve-6f58dcfc97-zv685\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") " pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:34.990289 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:34.990243    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:35.356007 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:35.355978    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"]
Nov 03 13:25:35.357625 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:25:35.357592    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d27b197_bee1_457b_9d07_017d98f1e01c.slice/crio-aa49206b2191c34659a35d41ab1c9dcfe4863f4ec1a2e0385473dfafffa94e2a WatchSource:0}: Error finding container aa49206b2191c34659a35d41ab1c9dcfe4863f4ec1a2e0385473dfafffa94e2a: Status 404 returned error can't find the container with id aa49206b2191c34659a35d41ab1c9dcfe4863f4ec1a2e0385473dfafffa94e2a
Nov 03 13:25:35.530047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:35.530007    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" event={"ID":"3d27b197-bee1-457b-9d07-017d98f1e01c","Type":"ContainerStarted","Data":"0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5"}
Nov 03 13:25:35.530047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:35.530049    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" event={"ID":"3d27b197-bee1-457b-9d07-017d98f1e01c","Type":"ContainerStarted","Data":"aa49206b2191c34659a35d41ab1c9dcfe4863f4ec1a2e0385473dfafffa94e2a"}
Nov 03 13:25:35.638586 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:35.638454    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" podStartSLOduration=1.638435668 podStartE2EDuration="1.638435668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:35.637333162 +0000 UTC m=+5677.208917097" watchObservedRunningTime="2026-11-03 13:25:35.638435668 +0000 UTC m=+5677.210019585"
Nov 03 13:25:35.990962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:35.990923    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:37.652124 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:25:37.652082    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:25:44.990684 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:44.990642    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:45.003381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:45.003352    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:25:45.595109 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:25:45.595070    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:26:07.651536 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:26:07.651442    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:26:19.380459 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.380425    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"]
Nov 03 13:26:19.384130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.384107    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.432512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.432471    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:26:19.432734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.432595    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:26:19.479020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.478966    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:26:19.479020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.479017    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:26:19.481101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.481078    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"]
Nov 03 13:26:19.528706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.528667    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.528947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.528806    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nht6v\" (UniqueName: \"kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.528947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.528847    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.629733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.629691    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nht6v\" (UniqueName: \"kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.629733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.629738    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.630050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.629845    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.630527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.630480    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.632385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.632367    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.684140 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.684107    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nht6v\" (UniqueName: \"kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v\") pod \"maas-api-revoke-sub-79137ed4-n4vhb\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:19.696620 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:19.696589    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:20.061958 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:20.061932    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"]
Nov 03 13:26:20.063876 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:26:20.063844    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17cb49c_9c2e_4328_948a_01b696107d3a.slice/crio-8726a55ad3a08694800fa69c1fd1a5cfc0db911c2630c16f0761b0431c439f58 WatchSource:0}: Error finding container 8726a55ad3a08694800fa69c1fd1a5cfc0db911c2630c16f0761b0431c439f58: Status 404 returned error can't find the container with id 8726a55ad3a08694800fa69c1fd1a5cfc0db911c2630c16f0761b0431c439f58
Nov 03 13:26:20.749088 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:20.749048    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb" event={"ID":"b17cb49c-9c2e-4328-948a-01b696107d3a","Type":"ContainerStarted","Data":"84ba3ca7514f397ce8d2ffa906f71caab8c8467ddfd7ddc16763861e1c354c0b"}
Nov 03 13:26:20.749538 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:20.749115    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb" event={"ID":"b17cb49c-9c2e-4328-948a-01b696107d3a","Type":"ContainerStarted","Data":"8726a55ad3a08694800fa69c1fd1a5cfc0db911c2630c16f0761b0431c439f58"}
Nov 03 13:26:20.857917 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:20.857861    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb" podStartSLOduration=1.857846731 podStartE2EDuration="1.857846731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:20.855954033 +0000 UTC m=+5722.427537946" watchObservedRunningTime="2026-11-03 13:26:20.857846731 +0000 UTC m=+5722.429430643"
Nov 03 13:26:21.755447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:21.755411    4546 generic.go:356] "Generic (PLEG): container finished" podID="b17cb49c-9c2e-4328-948a-01b696107d3a" containerID="84ba3ca7514f397ce8d2ffa906f71caab8c8467ddfd7ddc16763861e1c354c0b" exitCode=0
Nov 03 13:26:21.755898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:21.755488    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb" event={"ID":"b17cb49c-9c2e-4328-948a-01b696107d3a","Type":"ContainerDied","Data":"84ba3ca7514f397ce8d2ffa906f71caab8c8467ddfd7ddc16763861e1c354c0b"}
Nov 03 13:26:22.044519 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:22.044422    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"]
Nov 03 13:26:22.047560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:22.047530    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kvbnl"]
Nov 03 13:26:22.572520 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:22.572480    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b982ccaa-6710-4265-930a-9504e6954946" path="/var/lib/kubelet/pods/b982ccaa-6710-4265-930a-9504e6954946/volumes"
Nov 03 13:26:22.896855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:22.896827    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:23.063455 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.063417    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca\") pod \"b17cb49c-9c2e-4328-948a-01b696107d3a\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") "
Nov 03 13:26:23.063671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.063492    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v\" (UniqueName: \"kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v\") pod \"b17cb49c-9c2e-4328-948a-01b696107d3a\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") "
Nov 03 13:26:23.063671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.063516    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca\") pod \"b17cb49c-9c2e-4328-948a-01b696107d3a\" (UID: \"b17cb49c-9c2e-4328-948a-01b696107d3a\") "
Nov 03 13:26:23.063988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.063965    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca" pod "b17cb49c-9c2e-4328-948a-01b696107d3a" (UID: "b17cb49c-9c2e-4328-948a-01b696107d3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:23.065871 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.065850    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca" pod "b17cb49c-9c2e-4328-948a-01b696107d3a" (UID: "b17cb49c-9c2e-4328-948a-01b696107d3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:23.065977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.065951    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v" pod "b17cb49c-9c2e-4328-948a-01b696107d3a" (UID: "b17cb49c-9c2e-4328-948a-01b696107d3a"). InnerVolumeSpecName "kube-api-access-nht6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:23.164794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.164670    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:23.164794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.164703    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nht6v\" (UniqueName: \"kubernetes.io/projected/b17cb49c-9c2e-4328-948a-01b696107d3a-kube-api-access-nht6v\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:23.164794 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.164719    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17cb49c-9c2e-4328-948a-01b696107d3a-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:23.765943 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.765911    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb" event={"ID":"b17cb49c-9c2e-4328-948a-01b696107d3a","Type":"ContainerDied","Data":"8726a55ad3a08694800fa69c1fd1a5cfc0db911c2630c16f0761b0431c439f58"}
Nov 03 13:26:23.766145 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.765951    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"
Nov 03 13:26:23.766145 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:23.765959    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8726a55ad3a08694800fa69c1fd1a5cfc0db911c2630c16f0761b0431c439f58"
Nov 03 13:26:35.469830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:35.469790    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"]
Nov 03 13:26:37.657424 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:26:37.657381    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:26:50.490775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.490681    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" podUID="3d27b197-bee1-457b-9d07-017d98f1e01c" containerName="main" containerID="cri-o://0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5" gracePeriod=45
Nov 03 13:26:50.584994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.584948    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" podUID="3d27b197-bee1-457b-9d07-017d98f1e01c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.125:8000/ready\": dial tcp 10.133.1.125:8000: connect: connection refused"
Nov 03 13:26:50.748891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.748826    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:26:50.848798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.848737    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5\" (UniqueName: \"kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5\") pod \"3d27b197-bee1-457b-9d07-017d98f1e01c\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") "
Nov 03 13:26:50.848947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.848835    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir\") pod \"3d27b197-bee1-457b-9d07-017d98f1e01c\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") "
Nov 03 13:26:50.848947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.848896    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home\") pod \"3d27b197-bee1-457b-9d07-017d98f1e01c\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") "
Nov 03 13:26:50.848947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.848935    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs\") pod \"3d27b197-bee1-457b-9d07-017d98f1e01c\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") "
Nov 03 13:26:50.849090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.848990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm\") pod \"3d27b197-bee1-457b-9d07-017d98f1e01c\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") "
Nov 03 13:26:50.849090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.849069    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache\") pod \"3d27b197-bee1-457b-9d07-017d98f1e01c\" (UID: \"3d27b197-bee1-457b-9d07-017d98f1e01c\") "
Nov 03 13:26:50.849201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.849118    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir" pod "3d27b197-bee1-457b-9d07-017d98f1e01c" (UID: "3d27b197-bee1-457b-9d07-017d98f1e01c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:26:50.849271 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.849192    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home" pod "3d27b197-bee1-457b-9d07-017d98f1e01c" (UID: "3d27b197-bee1-457b-9d07-017d98f1e01c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:26:50.849394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.849369    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache" pod "3d27b197-bee1-457b-9d07-017d98f1e01c" (UID: "3d27b197-bee1-457b-9d07-017d98f1e01c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:26:50.849462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.849403    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:50.849462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.849417    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:50.851314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.851281    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm" pod "3d27b197-bee1-457b-9d07-017d98f1e01c" (UID: "3d27b197-bee1-457b-9d07-017d98f1e01c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:26:50.851428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.851336    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5" pod "3d27b197-bee1-457b-9d07-017d98f1e01c" (UID: "3d27b197-bee1-457b-9d07-017d98f1e01c"). InnerVolumeSpecName "kube-api-access-49dv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:50.851428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.851335    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs" pod "3d27b197-bee1-457b-9d07-017d98f1e01c" (UID: "3d27b197-bee1-457b-9d07-017d98f1e01c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:50.898466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.898431    4546 generic.go:356] "Generic (PLEG): container finished" podID="3d27b197-bee1-457b-9d07-017d98f1e01c" containerID="0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5" exitCode=0
Nov 03 13:26:50.898675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.898482    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"
Nov 03 13:26:50.898675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.898498    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" event={"ID":"3d27b197-bee1-457b-9d07-017d98f1e01c","Type":"ContainerDied","Data":"0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5"}
Nov 03 13:26:50.898675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.898557    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685" event={"ID":"3d27b197-bee1-457b-9d07-017d98f1e01c","Type":"ContainerDied","Data":"aa49206b2191c34659a35d41ab1c9dcfe4863f4ec1a2e0385473dfafffa94e2a"}
Nov 03 13:26:50.898675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.898583    4546 scope.go:122] "RemoveContainer" containerID="0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5"
Nov 03 13:26:50.911700 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.911675    4546 scope.go:122] "RemoveContainer" containerID="0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5"
Nov 03 13:26:50.912110 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:26:50.912089    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5\": container with ID starting with 0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5 not found: ID does not exist" containerID="0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5"
Nov 03 13:26:50.912215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.912119    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5"} err="failed to get container status \"0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5\": rpc error: code = NotFound desc = could not find container \"0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5\": container with ID starting with 0fc88444d1191d1132783c5046ca58a40720b395f5de8cfdeedacfa3354b03e5 not found: ID does not exist"
Nov 03 13:26:50.950609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.950569    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:50.950609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.950600    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dv5\" (UniqueName: \"kubernetes.io/projected/3d27b197-bee1-457b-9d07-017d98f1e01c-kube-api-access-49dv5\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:50.950609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.950616    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27b197-bee1-457b-9d07-017d98f1e01c-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:50.950913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:50.950629    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27b197-bee1-457b-9d07-017d98f1e01c-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:51.051337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:51.051235    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"]
Nov 03 13:26:51.055790 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:51.055762    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-55973f/simulator-single-main-55973f-kserve-6f58dcfc97-zv685"]
Nov 03 13:26:52.572076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:26:52.572042    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d27b197-bee1-457b-9d07-017d98f1e01c" path="/var/lib/kubelet/pods/3d27b197-bee1-457b-9d07-017d98f1e01c/volumes"
Nov 03 13:27:03.014356 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:03.014314    4546 scope.go:122] "RemoveContainer" containerID="5b481511bbac487ae90b412e6eedd7d9f1e723043fdb33d0d6dd2c8b4a2e2891"
Nov 03 13:27:07.651736 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:27:07.651690    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:27:12.043583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:12.043538    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"]
Nov 03 13:27:12.089290 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:12.089229    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-85g24"]
Nov 03 13:27:12.571796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:12.571742    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd8696a-c834-42ef-a0af-e393963a113f" path="/var/lib/kubelet/pods/2dd8696a-c834-42ef-a0af-e393963a113f/volumes"
Nov 03 13:27:31.879973 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.879933    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"]
Nov 03 13:27:31.880409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.880323    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3d27b197-bee1-457b-9d07-017d98f1e01c" containerName="main"
Nov 03 13:27:31.880409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.880335    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d27b197-bee1-457b-9d07-017d98f1e01c" containerName="main"
Nov 03 13:27:31.880409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.880344    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b17cb49c-9c2e-4328-948a-01b696107d3a" containerName="revoke-keys"
Nov 03 13:27:31.880409 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.880349    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17cb49c-9c2e-4328-948a-01b696107d3a" containerName="revoke-keys"
Nov 03 13:27:31.880554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.880434    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d27b197-bee1-457b-9d07-017d98f1e01c" containerName="main"
Nov 03 13:27:31.880554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.880445    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17cb49c-9c2e-4328-948a-01b696107d3a" containerName="revoke-keys"
Nov 03 13:27:31.883547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.883527    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:31.978377 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.978343    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9aad7fec-kserve-self-signed-certs\""
Nov 03 13:27:31.980828 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:31.980802    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"]
Nov 03 13:27:32.042580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.042542    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.042791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.042596    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.042791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.042677    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wtn\" (UniqueName: \"kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.042791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.042719    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.042919 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.042829    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.042919 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.042862    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144079    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144139    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144172    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wtn\" (UniqueName: \"kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144207    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144235    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144441 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144267    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144569    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144589    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.144817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.144720    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.146838 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.146814    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.146978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.146959    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.198639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.198603    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wtn\" (UniqueName: \"kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn\") pod \"e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") " pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.494526 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.494492    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:32.861079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:32.861052    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"]
Nov 03 13:27:32.863641 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:27:32.863604    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4deba928_4a79_42a6_8fe6_c0611160a197.slice/crio-a784a9a09c6e389f25a07d19bcfe6b1d68b8a1eb04434147b4647e212dd6596a WatchSource:0}: Error finding container a784a9a09c6e389f25a07d19bcfe6b1d68b8a1eb04434147b4647e212dd6596a: Status 404 returned error can't find the container with id a784a9a09c6e389f25a07d19bcfe6b1d68b8a1eb04434147b4647e212dd6596a
Nov 03 13:27:33.097228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:33.097184    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29" event={"ID":"4deba928-4a79-42a6-8fe6-c0611160a197","Type":"ContainerStarted","Data":"726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee"}
Nov 03 13:27:33.097228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:33.097232    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29" event={"ID":"4deba928-4a79-42a6-8fe6-c0611160a197","Type":"ContainerStarted","Data":"a784a9a09c6e389f25a07d19bcfe6b1d68b8a1eb04434147b4647e212dd6596a"}
Nov 03 13:27:33.206078 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:33.205967    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29" podStartSLOduration=2.205954346 podStartE2EDuration="2.205954346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:33.204157025 +0000 UTC m=+5794.775740958" watchObservedRunningTime="2026-11-03 13:27:33.205954346 +0000 UTC m=+5794.777538258"
Nov 03 13:27:33.495342 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:33.495307    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:37.658880 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:27:37.658783    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:27:42.494772 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:42.494729    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:42.507513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:42.507485    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:43.205290 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:43.205253    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:27:47.044574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:47.044533    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"]
Nov 03 13:27:47.091002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:47.090971    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-455zk"]
Nov 03 13:27:48.571537 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:27:48.571502    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a13ebf8-7fac-4f2b-89cd-31bea173bfd1" path="/var/lib/kubelet/pods/2a13ebf8-7fac-4f2b-89cd-31bea173bfd1/volumes"
Nov 03 13:28:03.070198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:03.070156    4546 scope.go:122] "RemoveContainer" containerID="e62b74d4259544531fb16f12b2d3312a67ef1e5a90eab48ab7b094c7842346fb"
Nov 03 13:28:06.042453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:06.042411    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"]
Nov 03 13:28:06.088378 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:06.088338    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2g22w"]
Nov 03 13:28:06.571328 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:06.571293    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d73d8d9-95a5-4181-a77e-c1ee90b573b1" path="/var/lib/kubelet/pods/9d73d8d9-95a5-4181-a77e-c1ee90b573b1/volumes"
Nov 03 13:28:07.651188 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:28:07.651150    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:28:18.052221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:18.052181    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"]
Nov 03 13:28:18.098205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:18.098170    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-skzht"]
Nov 03 13:28:18.571918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:18.571884    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9589c8b-b679-4c82-95be-2dfae0f22e43" path="/var/lib/kubelet/pods/f9589c8b-b679-4c82-95be-2dfae0f22e43/volumes"
Nov 03 13:28:25.135673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.135643    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"]
Nov 03 13:28:25.139235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.139215    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.189051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.189021    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:28:25.191137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.191115    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:28:25.191260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.191186    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:28:25.230004 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.229962    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zll2z\" (UniqueName: \"kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.230185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.230108    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.230185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.230173    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.231475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.231459    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:28:25.238424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.238396    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"]
Nov 03 13:28:25.331278 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.331232    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.331504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.331319    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zll2z\" (UniqueName: \"kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.331504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.331386    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.332143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.332122    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.333916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.333895    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.385795 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.385693    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zll2z\" (UniqueName: \"kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z\") pod \"maas-api-revoke-sub-452b6d7b-sw8t7\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.451849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.451809    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:25.843194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:25.843161    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"]
Nov 03 13:28:25.846914 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:28:25.846874    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92046aca_581f_47d8_bdaf_cc1786b9a797.slice/crio-951e79b4322eceb40863cb1503a21845d9324d22e9c486d514a4e9aa3884ba2b WatchSource:0}: Error finding container 951e79b4322eceb40863cb1503a21845d9324d22e9c486d514a4e9aa3884ba2b: Status 404 returned error can't find the container with id 951e79b4322eceb40863cb1503a21845d9324d22e9c486d514a4e9aa3884ba2b
Nov 03 13:28:26.411185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:26.411084    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7" event={"ID":"92046aca-581f-47d8-bdaf-cc1786b9a797","Type":"ContainerStarted","Data":"8b6c30c296f060aca31268abde64bbd3f760af9a4043ce782a516fb1bf53df1e"}
Nov 03 13:28:26.411185 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:26.411149    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7" event={"ID":"92046aca-581f-47d8-bdaf-cc1786b9a797","Type":"ContainerStarted","Data":"951e79b4322eceb40863cb1503a21845d9324d22e9c486d514a4e9aa3884ba2b"}
Nov 03 13:28:26.523056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:26.523001    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7" podStartSLOduration=1.52298651 podStartE2EDuration="1.52298651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:26.521907304 +0000 UTC m=+5848.093491216" watchObservedRunningTime="2026-11-03 13:28:26.52298651 +0000 UTC m=+5848.094570421"
Nov 03 13:28:27.420028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:27.419982    4546 generic.go:356] "Generic (PLEG): container finished" podID="92046aca-581f-47d8-bdaf-cc1786b9a797" containerID="8b6c30c296f060aca31268abde64bbd3f760af9a4043ce782a516fb1bf53df1e" exitCode=0
Nov 03 13:28:27.420443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:27.420072    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7" event={"ID":"92046aca-581f-47d8-bdaf-cc1786b9a797","Type":"ContainerDied","Data":"8b6c30c296f060aca31268abde64bbd3f760af9a4043ce782a516fb1bf53df1e"}
Nov 03 13:28:28.896057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.896018    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:28.991494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.991454    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca\") pod \"92046aca-581f-47d8-bdaf-cc1786b9a797\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") "
Nov 03 13:28:28.991893 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.991869    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z\" (UniqueName: \"kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z\") pod \"92046aca-581f-47d8-bdaf-cc1786b9a797\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") "
Nov 03 13:28:28.992079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.992061    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca\") pod \"92046aca-581f-47d8-bdaf-cc1786b9a797\" (UID: \"92046aca-581f-47d8-bdaf-cc1786b9a797\") "
Nov 03 13:28:28.992523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.992314    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca" pod "92046aca-581f-47d8-bdaf-cc1786b9a797" (UID: "92046aca-581f-47d8-bdaf-cc1786b9a797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:28.992523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.992496    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:28.994128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.994094    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z" pod "92046aca-581f-47d8-bdaf-cc1786b9a797" (UID: "92046aca-581f-47d8-bdaf-cc1786b9a797"). InnerVolumeSpecName "kube-api-access-zll2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:28.994417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:28.994397    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca" pod "92046aca-581f-47d8-bdaf-cc1786b9a797" (UID: "92046aca-581f-47d8-bdaf-cc1786b9a797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:29.094221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.093910    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92046aca-581f-47d8-bdaf-cc1786b9a797-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:29.094221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.093947    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zll2z\" (UniqueName: \"kubernetes.io/projected/92046aca-581f-47d8-bdaf-cc1786b9a797-kube-api-access-zll2z\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:29.419834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.415927    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:29.419834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.416031    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:29.474774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.474518    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7" event={"ID":"92046aca-581f-47d8-bdaf-cc1786b9a797","Type":"ContainerDied","Data":"951e79b4322eceb40863cb1503a21845d9324d22e9c486d514a4e9aa3884ba2b"}
Nov 03 13:28:29.474774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.474608    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="951e79b4322eceb40863cb1503a21845d9324d22e9c486d514a4e9aa3884ba2b"
Nov 03 13:28:29.474774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:29.474713    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"
Nov 03 13:28:37.650388 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:28:37.650350    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:28:37.867670 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:37.867636    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"]
Nov 03 13:28:44.422928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:44.422877    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:44.423665 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:44.423627    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:52.888444 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:52.888396    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29" podUID="4deba928-4a79-42a6-8fe6-c0611160a197" containerName="main" containerID="cri-o://726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee" gracePeriod=45
Nov 03 13:28:53.150871 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.150807    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:28:53.204882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.204847    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache\") pod \"4deba928-4a79-42a6-8fe6-c0611160a197\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") "
Nov 03 13:28:53.205060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.204895    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir\") pod \"4deba928-4a79-42a6-8fe6-c0611160a197\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") "
Nov 03 13:28:53.205060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.204938    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm\") pod \"4deba928-4a79-42a6-8fe6-c0611160a197\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") "
Nov 03 13:28:53.205060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.204967    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs\" (UniqueName: \"kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs\") pod \"4deba928-4a79-42a6-8fe6-c0611160a197\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") "
Nov 03 13:28:53.205060 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205044    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn\" (UniqueName: \"kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn\") pod \"4deba928-4a79-42a6-8fe6-c0611160a197\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") "
Nov 03 13:28:53.205305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205076    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home\") pod \"4deba928-4a79-42a6-8fe6-c0611160a197\" (UID: \"4deba928-4a79-42a6-8fe6-c0611160a197\") "
Nov 03 13:28:53.205305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205175    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir" pod "4deba928-4a79-42a6-8fe6-c0611160a197" (UID: "4deba928-4a79-42a6-8fe6-c0611160a197"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:28:53.205305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205182    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache" pod "4deba928-4a79-42a6-8fe6-c0611160a197" (UID: "4deba928-4a79-42a6-8fe6-c0611160a197"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:28:53.205463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205409    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:53.205463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205424    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:53.205463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.205418    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home" pod "4deba928-4a79-42a6-8fe6-c0611160a197" (UID: "4deba928-4a79-42a6-8fe6-c0611160a197"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:28:53.207319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.207294    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm" pod "4deba928-4a79-42a6-8fe6-c0611160a197" (UID: "4deba928-4a79-42a6-8fe6-c0611160a197"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:28:53.207319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.207308    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn" pod "4deba928-4a79-42a6-8fe6-c0611160a197" (UID: "4deba928-4a79-42a6-8fe6-c0611160a197"). InnerVolumeSpecName "kube-api-access-v6wtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:53.207475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.207354    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs" pod "4deba928-4a79-42a6-8fe6-c0611160a197" (UID: "4deba928-4a79-42a6-8fe6-c0611160a197"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:53.306727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.306697    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:53.306727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.306724    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4deba928-4a79-42a6-8fe6-c0611160a197-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:53.306963 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.306734    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wtn\" (UniqueName: \"kubernetes.io/projected/4deba928-4a79-42a6-8fe6-c0611160a197-kube-api-access-v6wtn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:53.306963 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.306762    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4deba928-4a79-42a6-8fe6-c0611160a197-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:53.470454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470411    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"]
Nov 03 13:28:53.470854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470839    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4deba928-4a79-42a6-8fe6-c0611160a197" containerName="main"
Nov 03 13:28:53.470903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470855    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deba928-4a79-42a6-8fe6-c0611160a197" containerName="main"
Nov 03 13:28:53.470903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470868    4546 cpu_manager.go:414] "containerMap: removing container" podUID="92046aca-581f-47d8-bdaf-cc1786b9a797" containerName="revoke-keys"
Nov 03 13:28:53.470903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470873    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92046aca-581f-47d8-bdaf-cc1786b9a797" containerName="revoke-keys"
Nov 03 13:28:53.471040 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470938    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deba928-4a79-42a6-8fe6-c0611160a197" containerName="main"
Nov 03 13:28:53.471040 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.470947    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="92046aca-581f-47d8-bdaf-cc1786b9a797" containerName="revoke-keys"
Nov 03 13:28:53.474269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.474248    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.508825 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.508788    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.509054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.509023    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.509126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.509081    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgfv\" (UniqueName: \"kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.522785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.522737    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:28:53.522967 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.522911    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:28:53.523038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.522982    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:28:53.523038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.522995    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:28:53.529831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.529807    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"]
Nov 03 13:28:53.609830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.609795    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.609830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.609837    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgfv\" (UniqueName: \"kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.610066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.609877    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.610445 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.610420    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.612385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.612363    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.647554 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.647520    4546 generic.go:356] "Generic (PLEG): container finished" podID="4deba928-4a79-42a6-8fe6-c0611160a197" containerID="726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee" exitCode=0
Nov 03 13:28:53.647697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.647570    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"
Nov 03 13:28:53.647697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.647589    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29" event={"ID":"4deba928-4a79-42a6-8fe6-c0611160a197","Type":"ContainerDied","Data":"726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee"}
Nov 03 13:28:53.647697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.647632    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29" event={"ID":"4deba928-4a79-42a6-8fe6-c0611160a197","Type":"ContainerDied","Data":"a784a9a09c6e389f25a07d19bcfe6b1d68b8a1eb04434147b4647e212dd6596a"}
Nov 03 13:28:53.647697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.647657    4546 scope.go:122] "RemoveContainer" containerID="726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee"
Nov 03 13:28:53.656533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.656519    4546 scope.go:122] "RemoveContainer" containerID="726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee"
Nov 03 13:28:53.656831 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:28:53.656810    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee\": container with ID starting with 726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee not found: ID does not exist" containerID="726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee"
Nov 03 13:28:53.656892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.656843    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee"} err="failed to get container status \"726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee\": rpc error: code = NotFound desc = could not find container \"726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee\": container with ID starting with 726043bbdc2cdaed1303258c1238b36257cb5658e87f1ec55e449ec14efebeee not found: ID does not exist"
Nov 03 13:28:53.662944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.662925    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgfv\" (UniqueName: \"kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv\") pod \"maas-api-revoke-sub-3e31a506-w5jnq\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.790486 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.790398    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:53.800257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.800227    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"]
Nov 03 13:28:53.803319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:53.803290    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9aad7fec-kserve-867cc884bf-57f29"]
Nov 03 13:28:54.153055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:54.153012    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"]
Nov 03 13:28:54.157828 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:28:54.157797    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d399602_0b3d_40e3_81f4_81704d54298a.slice/crio-7dc6c1a3cafc58f866424782ca868bd436e960668e3b1d1f0c52e363151b2bdc WatchSource:0}: Error finding container 7dc6c1a3cafc58f866424782ca868bd436e960668e3b1d1f0c52e363151b2bdc: Status 404 returned error can't find the container with id 7dc6c1a3cafc58f866424782ca868bd436e960668e3b1d1f0c52e363151b2bdc
Nov 03 13:28:54.571860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:54.571818    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deba928-4a79-42a6-8fe6-c0611160a197" path="/var/lib/kubelet/pods/4deba928-4a79-42a6-8fe6-c0611160a197/volumes"
Nov 03 13:28:54.656122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:54.656072    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq" event={"ID":"0d399602-0b3d-40e3-81f4-81704d54298a","Type":"ContainerStarted","Data":"c5bd0e56df46fb0334bdee510eed785a49f18fa2183e16aef1f0bf273b246bda"}
Nov 03 13:28:54.656357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:54.656194    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq" event={"ID":"0d399602-0b3d-40e3-81f4-81704d54298a","Type":"ContainerStarted","Data":"7dc6c1a3cafc58f866424782ca868bd436e960668e3b1d1f0c52e363151b2bdc"}
Nov 03 13:28:54.763857 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:54.763804    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq" podStartSLOduration=1.763789168 podStartE2EDuration="1.763789168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:54.762845703 +0000 UTC m=+5876.334429615" watchObservedRunningTime="2026-11-03 13:28:54.763789168 +0000 UTC m=+5876.335373079"
Nov 03 13:28:55.663706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:55.663670    4546 generic.go:356] "Generic (PLEG): container finished" podID="0d399602-0b3d-40e3-81f4-81704d54298a" containerID="c5bd0e56df46fb0334bdee510eed785a49f18fa2183e16aef1f0bf273b246bda" exitCode=0
Nov 03 13:28:55.663706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:55.663710    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq" event={"ID":"0d399602-0b3d-40e3-81f4-81704d54298a","Type":"ContainerDied","Data":"c5bd0e56df46fb0334bdee510eed785a49f18fa2183e16aef1f0bf273b246bda"}
Nov 03 13:28:56.807962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.807937    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:56.839734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.839699    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv\" (UniqueName: \"kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv\") pod \"0d399602-0b3d-40e3-81f4-81704d54298a\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") "
Nov 03 13:28:56.839734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.839737    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca\") pod \"0d399602-0b3d-40e3-81f4-81704d54298a\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") "
Nov 03 13:28:56.840064 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.839829    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca\") pod \"0d399602-0b3d-40e3-81f4-81704d54298a\" (UID: \"0d399602-0b3d-40e3-81f4-81704d54298a\") "
Nov 03 13:28:56.840138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.840091    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca" pod "0d399602-0b3d-40e3-81f4-81704d54298a" (UID: "0d399602-0b3d-40e3-81f4-81704d54298a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:56.840239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.840218    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:56.842257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.842223    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca" pod "0d399602-0b3d-40e3-81f4-81704d54298a" (UID: "0d399602-0b3d-40e3-81f4-81704d54298a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:56.842395 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.842284    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv" pod "0d399602-0b3d-40e3-81f4-81704d54298a" (UID: "0d399602-0b3d-40e3-81f4-81704d54298a"). InnerVolumeSpecName "kube-api-access-fkgfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:56.941656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.941571    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgfv\" (UniqueName: \"kubernetes.io/projected/0d399602-0b3d-40e3-81f4-81704d54298a-kube-api-access-fkgfv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:56.941656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:56.941603    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d399602-0b3d-40e3-81f4-81704d54298a-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:57.676806 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:57.676766    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq" event={"ID":"0d399602-0b3d-40e3-81f4-81704d54298a","Type":"ContainerDied","Data":"7dc6c1a3cafc58f866424782ca868bd436e960668e3b1d1f0c52e363151b2bdc"}
Nov 03 13:28:57.677041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:57.676823    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dc6c1a3cafc58f866424782ca868bd436e960668e3b1d1f0c52e363151b2bdc"
Nov 03 13:28:57.677041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:57.676787    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"
Nov 03 13:28:59.422781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:59.419888    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:59.422781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:28:59.419985    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:29:03.097989 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:03.097958    4546 scope.go:122] "RemoveContainer" containerID="4d6b3d53dbc07ef8864cb72e1845aa52139fcdd41da733be6e33a37371dd9e55"
Nov 03 13:29:03.106932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:03.106907    4546 scope.go:122] "RemoveContainer" containerID="ceb4d408a9dc17064c284a06fdcc3e044662311536aad34bff8acf1243bfec99"
Nov 03 13:29:07.650957 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:29:07.650874    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:29:14.420941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:14.420880    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:29:14.421646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:14.421592    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:29:14.421929 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:14.421910    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:29:15.041770 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:15.041708    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc"
Nov 03 13:29:18.574058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.574025    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"]
Nov 03 13:29:18.574464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.574379    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0d399602-0b3d-40e3-81f4-81704d54298a" containerName="revoke-keys"
Nov 03 13:29:18.574464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.574390    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d399602-0b3d-40e3-81f4-81704d54298a" containerName="revoke-keys"
Nov 03 13:29:18.574464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.574463    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d399602-0b3d-40e3-81f4-81704d54298a" containerName="revoke-keys"
Nov 03 13:29:18.577463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.577444    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.626612 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.626579    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"]
Nov 03 13:29:18.669838 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.669800    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:29:18.670029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.669978    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:29:18.670119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.670101    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:29:18.672030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.672018    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:29:18.701891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.701845    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n24z\" (UniqueName: \"kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.702081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.701904    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.702081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.701985    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.802803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.802740    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n24z\" (UniqueName: \"kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.802997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.802823    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.802997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.802848    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.803500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.803478    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.805447 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.805428    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.856155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.856075    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n24z\" (UniqueName: \"kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z\") pod \"maas-api-revoke-sub-9fecd4b7-rx6br\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:18.890952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:18.890913    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:19.286085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:19.286047    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"]
Nov 03 13:29:19.289766 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:29:19.289720    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6c2e08_70e1_4652_a523_544a355d6abd.slice/crio-4312aeeefe670d474fd4939851b97156a30d215f2de51c8d3255ce0545bd337d WatchSource:0}: Error finding container 4312aeeefe670d474fd4939851b97156a30d215f2de51c8d3255ce0545bd337d: Status 404 returned error can't find the container with id 4312aeeefe670d474fd4939851b97156a30d215f2de51c8d3255ce0545bd337d
Nov 03 13:29:19.807764 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:19.807716    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br" event={"ID":"5f6c2e08-70e1-4652-a523-544a355d6abd","Type":"ContainerStarted","Data":"d4e59782e9d05e267638a2f2e24e5d7b0a72abda1a105cea27be2ead914b50b3"}
Nov 03 13:29:19.808263 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:19.807800    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br" event={"ID":"5f6c2e08-70e1-4652-a523-544a355d6abd","Type":"ContainerStarted","Data":"4312aeeefe670d474fd4939851b97156a30d215f2de51c8d3255ce0545bd337d"}
Nov 03 13:29:19.914331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:19.914225    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br" podStartSLOduration=1.914211898 podStartE2EDuration="1.914211898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:19.913093184 +0000 UTC m=+5901.484677108" watchObservedRunningTime="2026-11-03 13:29:19.914211898 +0000 UTC m=+5901.485795809"
Nov 03 13:29:20.813493 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:20.813459    4546 generic.go:356] "Generic (PLEG): container finished" podID="5f6c2e08-70e1-4652-a523-544a355d6abd" containerID="d4e59782e9d05e267638a2f2e24e5d7b0a72abda1a105cea27be2ead914b50b3" exitCode=0
Nov 03 13:29:20.813948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:20.813550    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br" event={"ID":"5f6c2e08-70e1-4652-a523-544a355d6abd","Type":"ContainerDied","Data":"d4e59782e9d05e267638a2f2e24e5d7b0a72abda1a105cea27be2ead914b50b3"}
Nov 03 13:29:21.964186 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:21.964161    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:22.134812 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.134688    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z\" (UniqueName: \"kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z\") pod \"5f6c2e08-70e1-4652-a523-544a355d6abd\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") "
Nov 03 13:29:22.134812 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.134741    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca\") pod \"5f6c2e08-70e1-4652-a523-544a355d6abd\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") "
Nov 03 13:29:22.135091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.134820    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca\") pod \"5f6c2e08-70e1-4652-a523-544a355d6abd\" (UID: \"5f6c2e08-70e1-4652-a523-544a355d6abd\") "
Nov 03 13:29:22.135160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.135114    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca" pod "5f6c2e08-70e1-4652-a523-544a355d6abd" (UID: "5f6c2e08-70e1-4652-a523-544a355d6abd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:22.137038 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.137011    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z" pod "5f6c2e08-70e1-4652-a523-544a355d6abd" (UID: "5f6c2e08-70e1-4652-a523-544a355d6abd"). InnerVolumeSpecName "kube-api-access-8n24z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:22.137143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.137057    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca" pod "5f6c2e08-70e1-4652-a523-544a355d6abd" (UID: "5f6c2e08-70e1-4652-a523-544a355d6abd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:29:22.236345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.236315    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n24z\" (UniqueName: \"kubernetes.io/projected/5f6c2e08-70e1-4652-a523-544a355d6abd-kube-api-access-8n24z\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:22.236345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.236342    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:22.236345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.236352    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6c2e08-70e1-4652-a523-544a355d6abd-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:22.824710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.824676    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br" event={"ID":"5f6c2e08-70e1-4652-a523-544a355d6abd","Type":"ContainerDied","Data":"4312aeeefe670d474fd4939851b97156a30d215f2de51c8d3255ce0545bd337d"}
Nov 03 13:29:22.824710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.824702    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"
Nov 03 13:29:22.824990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:22.824736    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4312aeeefe670d474fd4939851b97156a30d215f2de51c8d3255ce0545bd337d"
Nov 03 13:29:35.405797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.405734    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-575967589b-czdmr"]
Nov 03 13:29:35.406552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.406153    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5f6c2e08-70e1-4652-a523-544a355d6abd" containerName="revoke-keys"
Nov 03 13:29:35.406552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.406164    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6c2e08-70e1-4652-a523-544a355d6abd" containerName="revoke-keys"
Nov 03 13:29:35.406552 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.406256    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6c2e08-70e1-4652-a523-544a355d6abd" containerName="revoke-keys"
Nov 03 13:29:35.409766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.409720    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.465463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.465418    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-575967589b-czdmr"]
Nov 03 13:29:35.467892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.467859    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume\") pod \"payload-pre-processing-575967589b-czdmr\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") " pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.468076 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.467905    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tz7\" (UniqueName: \"kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7\") pod \"payload-pre-processing-575967589b-czdmr\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") " pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.568879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.568834    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume\") pod \"payload-pre-processing-575967589b-czdmr\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") " pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.568879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.568872    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tz7\" (UniqueName: \"kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7\") pod \"payload-pre-processing-575967589b-czdmr\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") " pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.569595 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.569568    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume\") pod \"payload-pre-processing-575967589b-czdmr\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") " pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.622627 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.622585    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tz7\" (UniqueName: \"kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7\") pod \"payload-pre-processing-575967589b-czdmr\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") " pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:35.722208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:35.722179    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:36.083866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:36.083831    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-575967589b-czdmr"]
Nov 03 13:29:36.086005 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:29:36.085972    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51db9208_902f_4007_ab73_9746aa59fc43.slice/crio-f9abb095bb1f79e1a8cd2f63548d4f3d3dc0297ee72c941a8641a774ca0eb667 WatchSource:0}: Error finding container f9abb095bb1f79e1a8cd2f63548d4f3d3dc0297ee72c941a8641a774ca0eb667: Status 404 returned error can't find the container with id f9abb095bb1f79e1a8cd2f63548d4f3d3dc0297ee72c941a8641a774ca0eb667
Nov 03 13:29:36.896831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:36.896788    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr" event={"ID":"51db9208-902f-4007-ab73-9746aa59fc43","Type":"ContainerStarted","Data":"43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d"}
Nov 03 13:29:36.897322 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:36.896859    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr" event={"ID":"51db9208-902f-4007-ab73-9746aa59fc43","Type":"ContainerStarted","Data":"f9abb095bb1f79e1a8cd2f63548d4f3d3dc0297ee72c941a8641a774ca0eb667"}
Nov 03 13:29:36.897322 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:36.896930    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:37.005184 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:37.005121    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr" podStartSLOduration=2.005106634 podStartE2EDuration="2.005106634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:37.002998674 +0000 UTC m=+5918.574582597" watchObservedRunningTime="2026-11-03 13:29:37.005106634 +0000 UTC m=+5918.576690546"
Nov 03 13:29:37.650843 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:29:37.650804    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:29:47.906029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:47.905997    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:29:48.062096 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:48.062053    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"]
Nov 03 13:29:50.027436 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.027396    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-674476bcfd-j7czc"]
Nov 03 13:29:50.031079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.031054    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.085439 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.085396    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-674476bcfd-j7czc"]
Nov 03 13:29:50.116225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.116180    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fplwd\" (UniqueName: \"kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd\") pod \"payload-processing-674476bcfd-j7czc\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") " pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.116405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.116256    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume\") pod \"payload-processing-674476bcfd-j7czc\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") " pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.217515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.217481    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fplwd\" (UniqueName: \"kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd\") pod \"payload-processing-674476bcfd-j7czc\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") " pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.217515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.217524    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume\") pod \"payload-processing-674476bcfd-j7czc\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") " pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.218333 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.218312    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume\") pod \"payload-processing-674476bcfd-j7czc\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") " pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.271417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.271377    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fplwd\" (UniqueName: \"kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd\") pod \"payload-processing-674476bcfd-j7czc\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") " pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.343654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.343559    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:50.707630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.707602    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-674476bcfd-j7czc"]
Nov 03 13:29:50.709923 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:29:50.709886    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679235d6_ea88_419c_85b5_74e8999f9958.slice/crio-fa96e2d8f0baf7f6acc875b5a65aed61d4ee5ec489130977da3e89bcea177cc0 WatchSource:0}: Error finding container fa96e2d8f0baf7f6acc875b5a65aed61d4ee5ec489130977da3e89bcea177cc0: Status 404 returned error can't find the container with id fa96e2d8f0baf7f6acc875b5a65aed61d4ee5ec489130977da3e89bcea177cc0
Nov 03 13:29:50.966180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.966150    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-674476bcfd-j7czc" event={"ID":"679235d6-ea88-419c-85b5-74e8999f9958","Type":"ContainerStarted","Data":"5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467"}
Nov 03 13:29:50.966371 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.966200    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-674476bcfd-j7czc" event={"ID":"679235d6-ea88-419c-85b5-74e8999f9958","Type":"ContainerStarted","Data":"fa96e2d8f0baf7f6acc875b5a65aed61d4ee5ec489130977da3e89bcea177cc0"}
Nov 03 13:29:50.966371 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:50.966236    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:29:51.074265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:51.074123    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-674476bcfd-j7czc" podStartSLOduration=2.074105059 podStartE2EDuration="2.074105059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:51.072479728 +0000 UTC m=+5932.644063653" watchObservedRunningTime="2026-11-03 13:29:51.074105059 +0000 UTC m=+5932.645688972"
Nov 03 13:29:53.063321 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.063274    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4" podUID="58ff9dfe-4265-4629-9aba-b7cb5ccd539f" containerName="payload-pre-processing" containerID="cri-o://d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2" gracePeriod=25
Nov 03 13:29:53.313915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.313840    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 13:29:53.448567 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.448526    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume\") pod \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") "
Nov 03 13:29:53.448769 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.448647    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh\" (UniqueName: \"kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh\") pod \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\" (UID: \"58ff9dfe-4265-4629-9aba-b7cb5ccd539f\") "
Nov 03 13:29:53.449047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.449018    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume" pod "58ff9dfe-4265-4629-9aba-b7cb5ccd539f" (UID: "58ff9dfe-4265-4629-9aba-b7cb5ccd539f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:53.449152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.449133    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.451066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.451039    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh" pod "58ff9dfe-4265-4629-9aba-b7cb5ccd539f" (UID: "58ff9dfe-4265-4629-9aba-b7cb5ccd539f"). InnerVolumeSpecName "kube-api-access-mxkrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:53.554693 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.550399    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxkrh\" (UniqueName: \"kubernetes.io/projected/58ff9dfe-4265-4629-9aba-b7cb5ccd539f-kube-api-access-mxkrh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.981237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.981201    4546 generic.go:356] "Generic (PLEG): container finished" podID="58ff9dfe-4265-4629-9aba-b7cb5ccd539f" containerID="d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2" exitCode=0
Nov 03 13:29:53.981438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.981258    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4" event={"ID":"58ff9dfe-4265-4629-9aba-b7cb5ccd539f","Type":"ContainerDied","Data":"d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2"}
Nov 03 13:29:53.981438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.981261    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"
Nov 03 13:29:53.981438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.981298    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4" event={"ID":"58ff9dfe-4265-4629-9aba-b7cb5ccd539f","Type":"ContainerDied","Data":"9dc26df55dd449601a50ebc628a0cec791a57fadbc49e95bc5490505dadcbe5f"}
Nov 03 13:29:53.981438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.981320    4546 scope.go:122] "RemoveContainer" containerID="d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2"
Nov 03 13:29:53.990737 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.990714    4546 scope.go:122] "RemoveContainer" containerID="d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2"
Nov 03 13:29:53.991076 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:29:53.991054    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2\": container with ID starting with d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2 not found: ID does not exist" containerID="d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2"
Nov 03 13:29:53.991131 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:53.991086    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2"} err="failed to get container status \"d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2\": rpc error: code = NotFound desc = could not find container \"d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2\": container with ID starting with d87c998ec00d1135cc673967178ae5a15fff8f569d830b8cf51c34303b47a9d2 not found: ID does not exist"
Nov 03 13:29:54.135403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:54.135370    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"]
Nov 03 13:29:54.181416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:54.181380    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kmmh4"]
Nov 03 13:29:54.571960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:29:54.571915    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ff9dfe-4265-4629-9aba-b7cb5ccd539f" path="/var/lib/kubelet/pods/58ff9dfe-4265-4629-9aba-b7cb5ccd539f/volumes"
Nov 03 13:30:00.159243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.159210    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"]
Nov 03 13:30:00.159797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.159615    4546 cpu_manager.go:414] "containerMap: removing container" podUID="58ff9dfe-4265-4629-9aba-b7cb5ccd539f" containerName="payload-pre-processing"
Nov 03 13:30:00.159797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.159627    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ff9dfe-4265-4629-9aba-b7cb5ccd539f" containerName="payload-pre-processing"
Nov 03 13:30:00.159797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.159689    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ff9dfe-4265-4629-9aba-b7cb5ccd539f" containerName="payload-pre-processing"
Nov 03 13:30:00.164572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.164545    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.215045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.215010    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:30:00.261005 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.260970    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:30:00.261200 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.261059    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:30:00.263134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.263111    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"]
Nov 03 13:30:00.320118 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.320080    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt69l\" (UniqueName: \"kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l\") pod \"maas-api-key-cleanup-29895210-m82q9\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.320330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.320158    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-m82q9\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.421833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.421718    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-m82q9\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.422026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.421845    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt69l\" (UniqueName: \"kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l\") pod \"maas-api-key-cleanup-29895210-m82q9\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.422430 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.422401    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-m82q9\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.475812 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.475777    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt69l\" (UniqueName: \"kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l\") pod \"maas-api-key-cleanup-29895210-m82q9\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.477604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.477586    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:00.845946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:00.845918    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"]
Nov 03 13:30:00.847980 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:30:00.847946    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efcf2f0_b413_42ec_ac19_2e309ddfb31e.slice/crio-987fa70f39ae7ef067bfb2eee38f39f7d92bc439e40f86bc571464577a15a44b WatchSource:0}: Error finding container 987fa70f39ae7ef067bfb2eee38f39f7d92bc439e40f86bc571464577a15a44b: Status 404 returned error can't find the container with id 987fa70f39ae7ef067bfb2eee38f39f7d92bc439e40f86bc571464577a15a44b
Nov 03 13:30:01.023214 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:01.023177    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" event={"ID":"8efcf2f0-b413-42ec-ac19-2e309ddfb31e","Type":"ContainerStarted","Data":"987fa70f39ae7ef067bfb2eee38f39f7d92bc439e40f86bc571464577a15a44b"}
Nov 03 13:30:01.974533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:01.974497    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:30:02.030296 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:02.030263    4546 generic.go:356] "Generic (PLEG): container finished" podID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerID="fa9ca1aa1f270ecdd56924cbb080590429248df100cedb253475a776f4a961d8" exitCode=60
Nov 03 13:30:02.030479 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:02.030365    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" event={"ID":"8efcf2f0-b413-42ec-ac19-2e309ddfb31e","Type":"ContainerDied","Data":"fa9ca1aa1f270ecdd56924cbb080590429248df100cedb253475a776f4a961d8"}
Nov 03 13:30:02.030570 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:02.030553    4546 scope.go:122] "RemoveContainer" containerID="fa9ca1aa1f270ecdd56924cbb080590429248df100cedb253475a776f4a961d8"
Nov 03 13:30:02.129466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:02.129434    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"]
Nov 03 13:30:03.036592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:03.036559    4546 generic.go:356] "Generic (PLEG): container finished" podID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerID="00106babcbfec83a00841dd5831b0b6beee3e2de6db86e9e05a0d49779e282d3" exitCode=60
Nov 03 13:30:03.037108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:03.036601    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" event={"ID":"8efcf2f0-b413-42ec-ac19-2e309ddfb31e","Type":"ContainerDied","Data":"00106babcbfec83a00841dd5831b0b6beee3e2de6db86e9e05a0d49779e282d3"}
Nov 03 13:30:03.037108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:03.036667    4546 scope.go:122] "RemoveContainer" containerID="fa9ca1aa1f270ecdd56924cbb080590429248df100cedb253475a776f4a961d8"
Nov 03 13:30:03.037108 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:03.036987    4546 scope.go:122] "RemoveContainer" containerID="00106babcbfec83a00841dd5831b0b6beee3e2de6db86e9e05a0d49779e282d3"
Nov 03 13:30:03.037272 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:03.037249    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-m82q9_odh-ai-gateway-infra(8efcf2f0-b413-42ec-ac19-2e309ddfb31e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e"
Nov 03 13:30:03.135686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:03.135600    4546 scope.go:122] "RemoveContainer" containerID="3f1e1a0430a172dc72833b039c417fdbd482ba9bf0dad6a52a2c3dcfbaa7e1e2"
Nov 03 13:30:03.144890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:03.144869    4546 scope.go:122] "RemoveContainer" containerID="be516a7c8dcaa8aba2dfc6b3c3762c987f92960bcdb80ee0c0a11ed5aec0a885"
Nov 03 13:30:04.044357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.044327    4546 scope.go:122] "RemoveContainer" containerID="00106babcbfec83a00841dd5831b0b6beee3e2de6db86e9e05a0d49779e282d3"
Nov 03 13:30:04.044802 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:04.044596    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-m82q9_odh-ai-gateway-infra(8efcf2f0-b413-42ec-ac19-2e309ddfb31e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e"
Nov 03 13:30:04.417688 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.417602    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"]
Nov 03 13:30:04.423023 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.423003    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.475678 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.475634    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"]
Nov 03 13:30:04.567269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.567229    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkw7k\" (UniqueName: \"kubernetes.io/projected/94894a65-6ffe-4944-9f4a-04c3c2e10489-kube-api-access-dkw7k\") pod \"payload-pre-processing-7c679d9f46-jk9ww\" (UID: \"94894a65-6ffe-4944-9f4a-04c3c2e10489\") " pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.567476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.567304    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94894a65-6ffe-4944-9f4a-04c3c2e10489-plugins-config-volume\") pod \"payload-pre-processing-7c679d9f46-jk9ww\" (UID: \"94894a65-6ffe-4944-9f4a-04c3c2e10489\") " pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.668100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.668008    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkw7k\" (UniqueName: \"kubernetes.io/projected/94894a65-6ffe-4944-9f4a-04c3c2e10489-kube-api-access-dkw7k\") pod \"payload-pre-processing-7c679d9f46-jk9ww\" (UID: \"94894a65-6ffe-4944-9f4a-04c3c2e10489\") " pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.668100 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.668093    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94894a65-6ffe-4944-9f4a-04c3c2e10489-plugins-config-volume\") pod \"payload-pre-processing-7c679d9f46-jk9ww\" (UID: \"94894a65-6ffe-4944-9f4a-04c3c2e10489\") " pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.668853 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.668828    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94894a65-6ffe-4944-9f4a-04c3c2e10489-plugins-config-volume\") pod \"payload-pre-processing-7c679d9f46-jk9ww\" (UID: \"94894a65-6ffe-4944-9f4a-04c3c2e10489\") " pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.721256 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.721223    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkw7k\" (UniqueName: \"kubernetes.io/projected/94894a65-6ffe-4944-9f4a-04c3c2e10489-kube-api-access-dkw7k\") pod \"payload-pre-processing-7c679d9f46-jk9ww\" (UID: \"94894a65-6ffe-4944-9f4a-04c3c2e10489\") " pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:04.736128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:04.736098    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:05.102342 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:05.102318    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"]
Nov 03 13:30:05.104863 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:30:05.104831    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94894a65_6ffe_4944_9f4a_04c3c2e10489.slice/crio-daaee1af79d1d582f6f4c11ff0bc34477085aa49ce6f348216799a5a47d9a9b6 WatchSource:0}: Error finding container daaee1af79d1d582f6f4c11ff0bc34477085aa49ce6f348216799a5a47d9a9b6: Status 404 returned error can't find the container with id daaee1af79d1d582f6f4c11ff0bc34477085aa49ce6f348216799a5a47d9a9b6
Nov 03 13:30:06.062626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:06.062579    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww" event={"ID":"94894a65-6ffe-4944-9f4a-04c3c2e10489","Type":"ContainerStarted","Data":"c7c36d090321fd2ed83d2e54fd032a0d30cc316848d26a70cc7ea49803925bdc"}
Nov 03 13:30:06.062822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:06.062672    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww" event={"ID":"94894a65-6ffe-4944-9f4a-04c3c2e10489","Type":"ContainerStarted","Data":"daaee1af79d1d582f6f4c11ff0bc34477085aa49ce6f348216799a5a47d9a9b6"}
Nov 03 13:30:06.062822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:06.062695    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:06.170165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:06.170090    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww" podStartSLOduration=2.170075059 podStartE2EDuration="2.170075059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:06.168331468 +0000 UTC m=+5947.739915580" watchObservedRunningTime="2026-11-03 13:30:06.170075059 +0000 UTC m=+5947.741659043"
Nov 03 13:30:07.130735 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.130690    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m" podUID="b10c10e5-e265-4bcf-b38a-1de049716594" containerName="payload-processing" containerID="cri-o://34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39" gracePeriod=25
Nov 03 13:30:07.382877 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.382814    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 13:30:07.391874 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.391852    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume\") pod \"b10c10e5-e265-4bcf-b38a-1de049716594\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") "
Nov 03 13:30:07.392081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.391925    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98\" (UniqueName: \"kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98\") pod \"b10c10e5-e265-4bcf-b38a-1de049716594\" (UID: \"b10c10e5-e265-4bcf-b38a-1de049716594\") "
Nov 03 13:30:07.392191 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.392167    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume" pod "b10c10e5-e265-4bcf-b38a-1de049716594" (UID: "b10c10e5-e265-4bcf-b38a-1de049716594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:07.394309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.394289    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98" pod "b10c10e5-e265-4bcf-b38a-1de049716594" (UID: "b10c10e5-e265-4bcf-b38a-1de049716594"). InnerVolumeSpecName "kube-api-access-lgf98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:07.492783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.492729    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10c10e5-e265-4bcf-b38a-1de049716594-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:07.492783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:07.492774    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgf98\" (UniqueName: \"kubernetes.io/projected/b10c10e5-e265-4bcf-b38a-1de049716594-kube-api-access-lgf98\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:07.650607 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:07.650517    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:30:08.074223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.074187    4546 generic.go:356] "Generic (PLEG): container finished" podID="b10c10e5-e265-4bcf-b38a-1de049716594" containerID="34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39" exitCode=0
Nov 03 13:30:08.074415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.074239    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"
Nov 03 13:30:08.074415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.074274    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m" event={"ID":"b10c10e5-e265-4bcf-b38a-1de049716594","Type":"ContainerDied","Data":"34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39"}
Nov 03 13:30:08.074415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.074335    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-zmp2m" event={"ID":"b10c10e5-e265-4bcf-b38a-1de049716594","Type":"ContainerDied","Data":"a2924f7ada07d659b56664c5c747731f366550ff0632ed6c287cdfffca5618f5"}
Nov 03 13:30:08.074415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.074359    4546 scope.go:122] "RemoveContainer" containerID="34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39"
Nov 03 13:30:08.084207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.084065    4546 scope.go:122] "RemoveContainer" containerID="34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39"
Nov 03 13:30:08.084450 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:08.084421    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39\": container with ID starting with 34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39 not found: ID does not exist" containerID="34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39"
Nov 03 13:30:08.084540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.084461    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39"} err="failed to get container status \"34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39\": rpc error: code = NotFound desc = could not find container \"34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39\": container with ID starting with 34df5ec8176d1fb05900607f31862038bb422cd26da84692a0c07ed29e4b2c39 not found: ID does not exist"
Nov 03 13:30:08.228696 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.228662    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"]
Nov 03 13:30:08.274903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.274866    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-zmp2m"]
Nov 03 13:30:08.572820 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:08.572788    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10c10e5-e265-4bcf-b38a-1de049716594" path="/var/lib/kubelet/pods/b10c10e5-e265-4bcf-b38a-1de049716594/volumes"
Nov 03 13:30:15.566561 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:15.566521    4546 scope.go:122] "RemoveContainer" containerID="00106babcbfec83a00841dd5831b0b6beee3e2de6db86e9e05a0d49779e282d3"
Nov 03 13:30:16.115916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:16.115827    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" event={"ID":"8efcf2f0-b413-42ec-ac19-2e309ddfb31e","Type":"ContainerStarted","Data":"f0ba8053d842d4a3f47899e75de6b1346a83da2a90b1f8b79b8c3aec0500d0a6"}
Nov 03 13:30:16.225942 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:16.225886    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" podStartSLOduration=16.225873947 podStartE2EDuration="16.225873947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:16.224141087 +0000 UTC m=+5957.795725016" watchObservedRunningTime="2026-11-03 13:30:16.225873947 +0000 UTC m=+5957.797457903"
Nov 03 13:30:16.692168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:16.692137    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-674476bcfd-j7czc"]
Nov 03 13:30:17.070960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.070928    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7c679d9f46-jk9ww"
Nov 03 13:30:17.122608 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.122576    4546 generic.go:356] "Generic (PLEG): container finished" podID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerID="f0ba8053d842d4a3f47899e75de6b1346a83da2a90b1f8b79b8c3aec0500d0a6" exitCode=60
Nov 03 13:30:17.122851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.122650    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" event={"ID":"8efcf2f0-b413-42ec-ac19-2e309ddfb31e","Type":"ContainerDied","Data":"f0ba8053d842d4a3f47899e75de6b1346a83da2a90b1f8b79b8c3aec0500d0a6"}
Nov 03 13:30:17.122851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.122717    4546 scope.go:122] "RemoveContainer" containerID="00106babcbfec83a00841dd5831b0b6beee3e2de6db86e9e05a0d49779e282d3"
Nov 03 13:30:17.123116 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.123099    4546 scope.go:122] "RemoveContainer" containerID="f0ba8053d842d4a3f47899e75de6b1346a83da2a90b1f8b79b8c3aec0500d0a6"
Nov 03 13:30:17.123383 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:17.123363    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-m82q9_odh-ai-gateway-infra(8efcf2f0-b413-42ec-ac19-2e309ddfb31e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e"
Nov 03 13:30:17.226325 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.226290    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-575967589b-czdmr"]
Nov 03 13:30:17.236252 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:17.236215    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"]
Nov 03 13:30:18.276428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.276400    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:18.402614 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.402516    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca\") pod \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") "
Nov 03 13:30:18.402614 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.402559    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l\" (UniqueName: \"kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l\") pod \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\" (UID: \"8efcf2f0-b413-42ec-ac19-2e309ddfb31e\") "
Nov 03 13:30:18.403077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.403044    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca" pod "8efcf2f0-b413-42ec-ac19-2e309ddfb31e" (UID: "8efcf2f0-b413-42ec-ac19-2e309ddfb31e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:18.405088 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.405062    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l" pod "8efcf2f0-b413-42ec-ac19-2e309ddfb31e" (UID: "8efcf2f0-b413-42ec-ac19-2e309ddfb31e"). InnerVolumeSpecName "kube-api-access-dt69l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:18.503525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.503487    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.503525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:18.503522    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt69l\" (UniqueName: \"kubernetes.io/projected/8efcf2f0-b413-42ec-ac19-2e309ddfb31e-kube-api-access-dt69l\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:19.135034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:19.135001    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"
Nov 03 13:30:19.135267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:19.135027    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9" event={"ID":"8efcf2f0-b413-42ec-ac19-2e309ddfb31e","Type":"ContainerDied","Data":"987fa70f39ae7ef067bfb2eee38f39f7d92bc439e40f86bc571464577a15a44b"}
Nov 03 13:30:19.135267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:19.135100    4546 scope.go:122] "RemoveContainer" containerID="f0ba8053d842d4a3f47899e75de6b1346a83da2a90b1f8b79b8c3aec0500d0a6"
Nov 03 13:30:19.288017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:19.287970    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"]
Nov 03 13:30:19.334109 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:19.334071    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-m82q9"]
Nov 03 13:30:20.572507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:20.572474    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" path="/var/lib/kubelet/pods/8efcf2f0-b413-42ec-ac19-2e309ddfb31e/volumes"
Nov 03 13:30:21.693429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:21.693374    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-674476bcfd-j7czc" podUID="679235d6-ea88-419c-85b5-74e8999f9958" containerName="payload-processing" containerID="cri-o://5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467" gracePeriod=25
Nov 03 13:30:21.946518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:21.946448    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:30:22.040404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.040368    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd\" (UniqueName: \"kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd\") pod \"679235d6-ea88-419c-85b5-74e8999f9958\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") "
Nov 03 13:30:22.040404 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.040404    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume\") pod \"679235d6-ea88-419c-85b5-74e8999f9958\" (UID: \"679235d6-ea88-419c-85b5-74e8999f9958\") "
Nov 03 13:30:22.040892 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.040870    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume" pod "679235d6-ea88-419c-85b5-74e8999f9958" (UID: "679235d6-ea88-419c-85b5-74e8999f9958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:22.042761 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.042720    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd" pod "679235d6-ea88-419c-85b5-74e8999f9958" (UID: "679235d6-ea88-419c-85b5-74e8999f9958"). InnerVolumeSpecName "kube-api-access-fplwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:22.141864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.141823    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fplwd\" (UniqueName: \"kubernetes.io/projected/679235d6-ea88-419c-85b5-74e8999f9958-kube-api-access-fplwd\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:22.141864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.141856    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235d6-ea88-419c-85b5-74e8999f9958-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:22.153104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.153071    4546 generic.go:356] "Generic (PLEG): container finished" podID="679235d6-ea88-419c-85b5-74e8999f9958" containerID="5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467" exitCode=0
Nov 03 13:30:22.153274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.153117    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-674476bcfd-j7czc"
Nov 03 13:30:22.153274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.153157    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-674476bcfd-j7czc" event={"ID":"679235d6-ea88-419c-85b5-74e8999f9958","Type":"ContainerDied","Data":"5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467"}
Nov 03 13:30:22.153274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.153211    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-674476bcfd-j7czc" event={"ID":"679235d6-ea88-419c-85b5-74e8999f9958","Type":"ContainerDied","Data":"fa96e2d8f0baf7f6acc875b5a65aed61d4ee5ec489130977da3e89bcea177cc0"}
Nov 03 13:30:22.153274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.153235    4546 scope.go:122] "RemoveContainer" containerID="5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467"
Nov 03 13:30:22.163157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.163122    4546 scope.go:122] "RemoveContainer" containerID="5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467"
Nov 03 13:30:22.163476 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:22.163454    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467\": container with ID starting with 5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467 not found: ID does not exist" containerID="5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467"
Nov 03 13:30:22.163571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.163483    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467"} err="failed to get container status \"5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467\": rpc error: code = NotFound desc = could not find container \"5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467\": container with ID starting with 5ac292c461bc3f3e291d63eb7c8b000e73ddbe2793a9bafd5901e3aeb8c04467 not found: ID does not exist"
Nov 03 13:30:22.227208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.227157    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr" podUID="51db9208-902f-4007-ab73-9746aa59fc43" containerName="payload-pre-processing" containerID="cri-o://43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d" gracePeriod=25
Nov 03 13:30:22.305304 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.305272    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-674476bcfd-j7czc"]
Nov 03 13:30:22.308717 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.308679    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-674476bcfd-j7czc"]
Nov 03 13:30:22.459142 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.459112    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:30:22.546535 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.546448    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume\") pod \"51db9208-902f-4007-ab73-9746aa59fc43\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") "
Nov 03 13:30:22.546717 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.546587    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7\" (UniqueName: \"kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7\") pod \"51db9208-902f-4007-ab73-9746aa59fc43\" (UID: \"51db9208-902f-4007-ab73-9746aa59fc43\") "
Nov 03 13:30:22.546905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.546882    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume" pod "51db9208-902f-4007-ab73-9746aa59fc43" (UID: "51db9208-902f-4007-ab73-9746aa59fc43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:22.548743 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.548723    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7" pod "51db9208-902f-4007-ab73-9746aa59fc43" (UID: "51db9208-902f-4007-ab73-9746aa59fc43"). InnerVolumeSpecName "kube-api-access-28tz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:22.583040 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.582998    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679235d6-ea88-419c-85b5-74e8999f9958" path="/var/lib/kubelet/pods/679235d6-ea88-419c-85b5-74e8999f9958/volumes"
Nov 03 13:30:22.648025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.647985    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51db9208-902f-4007-ab73-9746aa59fc43-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:22.648025 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:22.648015    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tz7\" (UniqueName: \"kubernetes.io/projected/51db9208-902f-4007-ab73-9746aa59fc43-kube-api-access-28tz7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:23.162487 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.162449    4546 generic.go:356] "Generic (PLEG): container finished" podID="51db9208-902f-4007-ab73-9746aa59fc43" containerID="43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d" exitCode=0
Nov 03 13:30:23.163003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.162504    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr" event={"ID":"51db9208-902f-4007-ab73-9746aa59fc43","Type":"ContainerDied","Data":"43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d"}
Nov 03 13:30:23.163003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.162523    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr"
Nov 03 13:30:23.163003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.162558    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-575967589b-czdmr" event={"ID":"51db9208-902f-4007-ab73-9746aa59fc43","Type":"ContainerDied","Data":"f9abb095bb1f79e1a8cd2f63548d4f3d3dc0297ee72c941a8641a774ca0eb667"}
Nov 03 13:30:23.163003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.162591    4546 scope.go:122] "RemoveContainer" containerID="43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d"
Nov 03 13:30:23.171581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.171560    4546 scope.go:122] "RemoveContainer" containerID="43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d"
Nov 03 13:30:23.171869 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:23.171845    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d\": container with ID starting with 43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d not found: ID does not exist" containerID="43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d"
Nov 03 13:30:23.171948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.171880    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d"} err="failed to get container status \"43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d\": rpc error: code = NotFound desc = could not find container \"43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d\": container with ID starting with 43b1a0540c9d9d74110326c97d5f8700091bfa4b86f472fd8931e95ea16b6d5d not found: ID does not exist"
Nov 03 13:30:23.314438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.314402    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-575967589b-czdmr"]
Nov 03 13:30:23.318366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:23.318336    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-575967589b-czdmr"]
Nov 03 13:30:24.028548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028512    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"]
Nov 03 13:30:24.028919 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028897    4546 cpu_manager.go:414] "containerMap: removing container" podUID="51db9208-902f-4007-ab73-9746aa59fc43" containerName="payload-pre-processing"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028920    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51db9208-902f-4007-ab73-9746aa59fc43" containerName="payload-pre-processing"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028944    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028949    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028959    4546 cpu_manager.go:414] "containerMap: removing container" podUID="679235d6-ea88-419c-85b5-74e8999f9958" containerName="payload-processing"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028964    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679235d6-ea88-419c-85b5-74e8999f9958" containerName="payload-processing"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028973    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b10c10e5-e265-4bcf-b38a-1de049716594" containerName="payload-processing"
Nov 03 13:30:24.028979 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028979    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10c10e5-e265-4bcf-b38a-1de049716594" containerName="payload-processing"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.028995    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029002    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029014    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029021    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029103    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029114    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="679235d6-ea88-419c-85b5-74e8999f9958" containerName="payload-processing"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029121    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10c10e5-e265-4bcf-b38a-1de049716594" containerName="payload-processing"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029127    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029132    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efcf2f0-b413-42ec-ac19-2e309ddfb31e" containerName="cleanup"
Nov 03 13:30:24.029205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.029139    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="51db9208-902f-4007-ab73-9746aa59fc43" containerName="payload-pre-processing"
Nov 03 13:30:24.033528 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.033506    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.082944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.082907    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 13:30:24.130117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.130071    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 13:30:24.130117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.130073    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 13:30:24.130117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.130073    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 13:30:24.130385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.130147    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2th4c\""
Nov 03 13:30:24.130385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.130108    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 13:30:24.132205 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.132185    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"]
Nov 03 13:30:24.163028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.162992    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.163028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.163026    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.163504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.163046    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5snv\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.163504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.163165    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.264000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.263962    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.264197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.264067    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.264197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.264088    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.264197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.264105    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5snv\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.264930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.264779    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.264930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.264828    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.266625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.266605    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.317003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.316934    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5snv\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv\") pod \"payload-processing-external-model-5bcd7997-tbx88\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.345956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.345924    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:24.571885 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.571807    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51db9208-902f-4007-ab73-9746aa59fc43" path="/var/lib/kubelet/pods/51db9208-902f-4007-ab73-9746aa59fc43/volumes"
Nov 03 13:30:24.713384 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:24.713359    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"]
Nov 03 13:30:24.715378 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:30:24.715345    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5793c485_995b_4a01_b966_5128a61197c1.slice/crio-2ea132718e351a0fc877058706cb9500d1353b36d81e64164ac469c7b7a5fbe6 WatchSource:0}: Error finding container 2ea132718e351a0fc877058706cb9500d1353b36d81e64164ac469c7b7a5fbe6: Status 404 returned error can't find the container with id 2ea132718e351a0fc877058706cb9500d1353b36d81e64164ac469c7b7a5fbe6
Nov 03 13:30:25.176029 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:25.175974    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88" event={"ID":"5793c485-995b-4a01-b966-5128a61197c1","Type":"ContainerStarted","Data":"ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391"}
Nov 03 13:30:25.176589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:25.176049    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88" event={"ID":"5793c485-995b-4a01-b966-5128a61197c1","Type":"ContainerStarted","Data":"2ea132718e351a0fc877058706cb9500d1353b36d81e64164ac469c7b7a5fbe6"}
Nov 03 13:30:25.176589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:25.176085    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:25.283774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:25.283680    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88" podStartSLOduration=2.283658782 podStartE2EDuration="2.283658782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:25.282418437 +0000 UTC m=+5966.854002350" watchObservedRunningTime="2026-11-03 13:30:25.283658782 +0000 UTC m=+5966.855242694"
Nov 03 13:30:27.492017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:27.491968    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xfwjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:30:27.492421 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:27.492040    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xfwjk" podUID="93052df7-c1d3-42d5-8d77-e96e0fd78881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:30:36.188859 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:30:36.188824    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:30:37.651178 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:30:37.651090    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:31:07.651911 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:31:07.651868    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:31:23.045055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:23.045019    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"]
Nov 03 13:31:23.091002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:23.090959    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79137ed4-n4vhb"]
Nov 03 13:31:24.571391 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:24.571356    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17cb49c-9c2e-4328-948a-01b696107d3a" path="/var/lib/kubelet/pods/b17cb49c-9c2e-4328-948a-01b696107d3a/volumes"
Nov 03 13:31:37.651478 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:31:37.651441    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:31:56.463052 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.463019    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"]
Nov 03 13:31:56.466848 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.466830    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.515608 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.515575    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:31:56.562003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.561965    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:31:56.562003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.561987    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:31:56.562235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.562042    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:31:56.564186 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.564167    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"]
Nov 03 13:31:56.580269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.580230    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.580476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.580277    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.580557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.580464    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9g4f\" (UniqueName: \"kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.681239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.681197    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.681239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.681242    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.681550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.681487    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9g4f\" (UniqueName: \"kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.682082 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.682054    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.683903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.683884    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.735491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.735460    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9g4f\" (UniqueName: \"kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f\") pod \"maas-api-revoke-sub-e2547622-brlzr\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:56.779510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:56.779472    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:57.143612 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:57.143589    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"]
Nov 03 13:31:57.145912 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:31:57.145877    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9eeb7c1_440e_40bb_9a1d_64f80ba4d29c.slice/crio-830332480bf48dce6d0a652bf41a34cb587c0149c47811aad4a17fab6ac94942 WatchSource:0}: Error finding container 830332480bf48dce6d0a652bf41a34cb587c0149c47811aad4a17fab6ac94942: Status 404 returned error can't find the container with id 830332480bf48dce6d0a652bf41a34cb587c0149c47811aad4a17fab6ac94942
Nov 03 13:31:57.620435 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:57.620402    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr" event={"ID":"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c","Type":"ContainerStarted","Data":"f4635f7fb8e00ad6c1c988d07768e2ec64a0a20bb8bc5a0b9c64165c5124d008"}
Nov 03 13:31:57.620882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:57.620458    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr" event={"ID":"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c","Type":"ContainerStarted","Data":"830332480bf48dce6d0a652bf41a34cb587c0149c47811aad4a17fab6ac94942"}
Nov 03 13:31:57.736412 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:57.736359    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr" podStartSLOduration=1.736343702 podStartE2EDuration="1.736343702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:57.733953676 +0000 UTC m=+6059.305537588" watchObservedRunningTime="2026-11-03 13:31:57.736343702 +0000 UTC m=+6059.307927613"
Nov 03 13:31:58.626878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:58.626846    4546 generic.go:356] "Generic (PLEG): container finished" podID="a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" containerID="f4635f7fb8e00ad6c1c988d07768e2ec64a0a20bb8bc5a0b9c64165c5124d008" exitCode=0
Nov 03 13:31:58.627343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:58.626940    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr" event={"ID":"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c","Type":"ContainerDied","Data":"f4635f7fb8e00ad6c1c988d07768e2ec64a0a20bb8bc5a0b9c64165c5124d008"}
Nov 03 13:31:59.764949 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.764923    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:31:59.813363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.813329    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca\") pod \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") "
Nov 03 13:31:59.813523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.813388    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca\") pod \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") "
Nov 03 13:31:59.813523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.813459    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f\" (UniqueName: \"kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f\") pod \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\" (UID: \"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c\") "
Nov 03 13:31:59.813797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.813741    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca" pod "a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" (UID: "a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:59.813921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.813902    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:59.815738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.815712    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca" pod "a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" (UID: "a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:31:59.815876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.815801    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f" pod "a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" (UID: "a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c"). InnerVolumeSpecName "kube-api-access-q9g4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:59.915032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.914941    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:59.915032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:31:59.914973    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9g4f\" (UniqueName: \"kubernetes.io/projected/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c-kube-api-access-q9g4f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:00.637970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:00.637932    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr" event={"ID":"a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c","Type":"ContainerDied","Data":"830332480bf48dce6d0a652bf41a34cb587c0149c47811aad4a17fab6ac94942"}
Nov 03 13:32:00.638156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:00.637982    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="830332480bf48dce6d0a652bf41a34cb587c0149c47811aad4a17fab6ac94942"
Nov 03 13:32:00.638156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:00.637982    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"
Nov 03 13:32:07.650972 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:32:07.650884    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:32:07.946732 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:07.946649    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"]
Nov 03 13:32:12.949899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:12.949843    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88" podUID="5793c485-995b-4a01-b966-5128a61197c1" containerName="payload-processing" containerID="cri-o://ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391" gracePeriod=25
Nov 03 13:32:13.338026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.337989    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:32:13.488569 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.488529    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume\") pod \"5793c485-995b-4a01-b966-5128a61197c1\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") "
Nov 03 13:32:13.488793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.488598    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials\") pod \"5793c485-995b-4a01-b966-5128a61197c1\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") "
Nov 03 13:32:13.488793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.488653    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv\") pod \"5793c485-995b-4a01-b966-5128a61197c1\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") "
Nov 03 13:32:13.488793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.488716    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume\") pod \"5793c485-995b-4a01-b966-5128a61197c1\" (UID: \"5793c485-995b-4a01-b966-5128a61197c1\") "
Nov 03 13:32:13.489042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.489003    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume" pod "5793c485-995b-4a01-b966-5128a61197c1" (UID: "5793c485-995b-4a01-b966-5128a61197c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:13.489254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.489223    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume" pod "5793c485-995b-4a01-b966-5128a61197c1" (UID: "5793c485-995b-4a01-b966-5128a61197c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:13.491013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.490993    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv" pod "5793c485-995b-4a01-b966-5128a61197c1" (UID: "5793c485-995b-4a01-b966-5128a61197c1"). InnerVolumeSpecName "kube-api-access-f5snv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:13.491099 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.491025    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials" pod "5793c485-995b-4a01-b966-5128a61197c1" (UID: "5793c485-995b-4a01-b966-5128a61197c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:13.590413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.590318    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:13.590413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.590353    4546 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-provider-credentials\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:13.590413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.590362    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5snv\" (UniqueName: \"kubernetes.io/projected/5793c485-995b-4a01-b966-5128a61197c1-kube-api-access-f5snv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:13.590413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.590370    4546 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5793c485-995b-4a01-b966-5128a61197c1-routing-overlay-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:13.701636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.701600    4546 generic.go:356] "Generic (PLEG): container finished" podID="5793c485-995b-4a01-b966-5128a61197c1" containerID="ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391" exitCode=0
Nov 03 13:32:13.701852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.701647    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"
Nov 03 13:32:13.701852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.701686    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88" event={"ID":"5793c485-995b-4a01-b966-5128a61197c1","Type":"ContainerDied","Data":"ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391"}
Nov 03 13:32:13.701852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.701741    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tbx88" event={"ID":"5793c485-995b-4a01-b966-5128a61197c1","Type":"ContainerDied","Data":"2ea132718e351a0fc877058706cb9500d1353b36d81e64164ac469c7b7a5fbe6"}
Nov 03 13:32:13.701852 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.701774    4546 scope.go:122] "RemoveContainer" containerID="ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391"
Nov 03 13:32:13.711286 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.711267    4546 scope.go:122] "RemoveContainer" containerID="ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391"
Nov 03 13:32:13.711608 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:32:13.711589    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391\": container with ID starting with ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391 not found: ID does not exist" containerID="ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391"
Nov 03 13:32:13.711667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.711619    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391"} err="failed to get container status \"ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391\": rpc error: code = NotFound desc = could not find container \"ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391\": container with ID starting with ed68da6768d3b43e8ba0e5b1e717232997e8b127b18061e6711590f5fce5f391 not found: ID does not exist"
Nov 03 13:32:13.856034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.855940    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"]
Nov 03 13:32:13.858881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:13.858848    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tbx88"]
Nov 03 13:32:14.571424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:14.571391    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5793c485-995b-4a01-b966-5128a61197c1" path="/var/lib/kubelet/pods/5793c485-995b-4a01-b966-5128a61197c1/volumes"
Nov 03 13:32:26.926683 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.926642    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"]
Nov 03 13:32:26.927129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.927072    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5793c485-995b-4a01-b966-5128a61197c1" containerName="payload-processing"
Nov 03 13:32:26.927129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.927084    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5793c485-995b-4a01-b966-5128a61197c1" containerName="payload-processing"
Nov 03 13:32:26.927129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.927110    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" containerName="revoke-keys"
Nov 03 13:32:26.927129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.927115    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" containerName="revoke-keys"
Nov 03 13:32:26.927279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.927165    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5793c485-995b-4a01-b966-5128a61197c1" containerName="payload-processing"
Nov 03 13:32:26.927279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.927179    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" containerName="revoke-keys"
Nov 03 13:32:26.930278 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.930255    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:26.983654 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.983621    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:32:26.984807 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:26.984783    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:32:27.029978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.029936    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:32:27.029978 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.029960    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:32:27.032393 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.032368    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.032547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.032402    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zngz\" (UniqueName: \"kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.032547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.032431    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.032912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.032882    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"]
Nov 03 13:32:27.133585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.133540    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.133585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.133590    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zngz\" (UniqueName: \"kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.133856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.133626    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.134389 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.134367    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.136115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.136085    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.187816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.187702    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zngz\" (UniqueName: \"kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz\") pod \"maas-api-revoke-sub-c1e271a9-xk2l6\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.246143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.246110    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:27.610731 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.610703    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"]
Nov 03 13:32:27.612799 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:32:27.612764    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c141f38_a6c2_49c7_bf8f_70cce0d3eaae.slice/crio-a1780f293f80adcd8f42fe594724a82c68124a5c7464da08f5eb4cceb63284da WatchSource:0}: Error finding container a1780f293f80adcd8f42fe594724a82c68124a5c7464da08f5eb4cceb63284da: Status 404 returned error can't find the container with id a1780f293f80adcd8f42fe594724a82c68124a5c7464da08f5eb4cceb63284da
Nov 03 13:32:27.784153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:27.784118    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6" event={"ID":"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae","Type":"ContainerStarted","Data":"a1780f293f80adcd8f42fe594724a82c68124a5c7464da08f5eb4cceb63284da"}
Nov 03 13:32:28.790541 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:28.790507    4546 generic.go:356] "Generic (PLEG): container finished" podID="8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" containerID="041a0a07c10b30730f4e33ae8f3a70660d2ba0ab760cb6e07780f4113390fc6d" exitCode=0
Nov 03 13:32:28.791000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:28.790572    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6" event={"ID":"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae","Type":"ContainerDied","Data":"041a0a07c10b30730f4e33ae8f3a70660d2ba0ab760cb6e07780f4113390fc6d"}
Nov 03 13:32:29.927370 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:29.927345    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:30.061854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.061764    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca\") pod \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") "
Nov 03 13:32:30.061854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.061843    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca\") pod \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") "
Nov 03 13:32:30.062055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.061871    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz\" (UniqueName: \"kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz\") pod \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\" (UID: \"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae\") "
Nov 03 13:32:30.062316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.062281    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca" pod "8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" (UID: "8c141f38-a6c2-49c7-bf8f-70cce0d3eaae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:30.064096 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.064061    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca" pod "8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" (UID: "8c141f38-a6c2-49c7-bf8f-70cce0d3eaae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:32:30.064207 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.064144    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz" pod "8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" (UID: "8c141f38-a6c2-49c7-bf8f-70cce0d3eaae"). InnerVolumeSpecName "kube-api-access-6zngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:30.162963 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.162924    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.162963 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.162955    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zngz\" (UniqueName: \"kubernetes.io/projected/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-kube-api-access-6zngz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.162963 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.162967    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.801560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.801525    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"
Nov 03 13:32:30.801801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.801560    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6" event={"ID":"8c141f38-a6c2-49c7-bf8f-70cce0d3eaae","Type":"ContainerDied","Data":"a1780f293f80adcd8f42fe594724a82c68124a5c7464da08f5eb4cceb63284da"}
Nov 03 13:32:30.801801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:30.801613    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1780f293f80adcd8f42fe594724a82c68124a5c7464da08f5eb4cceb63284da"
Nov 03 13:32:37.651917 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:32:37.651877    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:32:52.163389 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.163341    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"]
Nov 03 13:32:52.164084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.163741    4546 cpu_manager.go:414] "containerMap: removing container" podUID="8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" containerName="revoke-keys"
Nov 03 13:32:52.164084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.163773    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" containerName="revoke-keys"
Nov 03 13:32:52.164084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.163872    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" containerName="revoke-keys"
Nov 03 13:32:52.167145 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.167120    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.215768 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.215718    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:32:52.226874 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.226838    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qztc\" (UniqueName: \"kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.227085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.226927    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.227085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.226961    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.261964 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.261934    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:32:52.261964 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.261959    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:32:52.263870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.263847    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"]
Nov 03 13:32:52.316385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.316349    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:32:52.328174 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.328141    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qztc\" (UniqueName: \"kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.328307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.328178    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.328307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.328213    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.329017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.328996    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.330785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.330738    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.381827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.381795    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qztc\" (UniqueName: \"kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc\") pod \"maas-api-revoke-sub-2e1f5b5f-67rx8\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.480362 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.480329    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:52.843179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.843152    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"]
Nov 03 13:32:52.845092 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:32:52.845065    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3becd7_955b_43fe_8d60_15bdd31689bf.slice/crio-6ba434288a5d89832dd7562577fdcc773edf0e1f93ff1eb1d1cf697c0596eefb WatchSource:0}: Error finding container 6ba434288a5d89832dd7562577fdcc773edf0e1f93ff1eb1d1cf697c0596eefb: Status 404 returned error can't find the container with id 6ba434288a5d89832dd7562577fdcc773edf0e1f93ff1eb1d1cf697c0596eefb
Nov 03 13:32:52.930437 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:52.930387    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8" event={"ID":"bf3becd7-955b-43fe-8d60-15bdd31689bf","Type":"ContainerStarted","Data":"6ba434288a5d89832dd7562577fdcc773edf0e1f93ff1eb1d1cf697c0596eefb"}
Nov 03 13:32:53.936472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:53.936433    4546 generic.go:356] "Generic (PLEG): container finished" podID="bf3becd7-955b-43fe-8d60-15bdd31689bf" containerID="ac1cc6346a9259751e8392d9c1dc34bc471b1aeeab14e891afb69f40b5522f9a" exitCode=0
Nov 03 13:32:53.937004 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:53.936485    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8" event={"ID":"bf3becd7-955b-43fe-8d60-15bdd31689bf","Type":"ContainerDied","Data":"ac1cc6346a9259751e8392d9c1dc34bc471b1aeeab14e891afb69f40b5522f9a"}
Nov 03 13:32:55.084936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.084906    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:55.156024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.155986    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc\" (UniqueName: \"kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc\") pod \"bf3becd7-955b-43fe-8d60-15bdd31689bf\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") "
Nov 03 13:32:55.156024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.156027    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca\") pod \"bf3becd7-955b-43fe-8d60-15bdd31689bf\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") "
Nov 03 13:32:55.156307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.156062    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca\") pod \"bf3becd7-955b-43fe-8d60-15bdd31689bf\" (UID: \"bf3becd7-955b-43fe-8d60-15bdd31689bf\") "
Nov 03 13:32:55.156449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.156427    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca" pod "bf3becd7-955b-43fe-8d60-15bdd31689bf" (UID: "bf3becd7-955b-43fe-8d60-15bdd31689bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:55.158462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.158433    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca" pod "bf3becd7-955b-43fe-8d60-15bdd31689bf" (UID: "bf3becd7-955b-43fe-8d60-15bdd31689bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:32:55.158588 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.158470    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc" pod "bf3becd7-955b-43fe-8d60-15bdd31689bf" (UID: "bf3becd7-955b-43fe-8d60-15bdd31689bf"). InnerVolumeSpecName "kube-api-access-7qztc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:55.257122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.257080    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qztc\" (UniqueName: \"kubernetes.io/projected/bf3becd7-955b-43fe-8d60-15bdd31689bf-kube-api-access-7qztc\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:55.257122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.257116    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:55.257122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.257127    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3becd7-955b-43fe-8d60-15bdd31689bf-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:55.947144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.947107    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8" event={"ID":"bf3becd7-955b-43fe-8d60-15bdd31689bf","Type":"ContainerDied","Data":"6ba434288a5d89832dd7562577fdcc773edf0e1f93ff1eb1d1cf697c0596eefb"}
Nov 03 13:32:55.947144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.947129    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"
Nov 03 13:32:55.947380 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:32:55.947157    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ba434288a5d89832dd7562577fdcc773edf0e1f93ff1eb1d1cf697c0596eefb"
Nov 03 13:33:03.203706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:03.203668    4546 scope.go:122] "RemoveContainer" containerID="84ba3ca7514f397ce8d2ffa906f71caab8c8467ddfd7ddc16763861e1c354c0b"
Nov 03 13:33:04.523913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.523880    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"]
Nov 03 13:33:04.524316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.524278    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bf3becd7-955b-43fe-8d60-15bdd31689bf" containerName="revoke-keys"
Nov 03 13:33:04.524316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.524290    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3becd7-955b-43fe-8d60-15bdd31689bf" containerName="revoke-keys"
Nov 03 13:33:04.524391 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.524360    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3becd7-955b-43fe-8d60-15bdd31689bf" containerName="revoke-keys"
Nov 03 13:33:04.528976 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.528953    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.625481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.625450    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:33:04.625481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.625467    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:33:04.625481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.625459    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:33:04.625481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.625463    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:33:04.627467 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.627447    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"]
Nov 03 13:33:04.657660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.657622    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwvz\" (UniqueName: \"kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.657863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.657781    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.657863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.657841    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.758730 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.758685    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwvz\" (UniqueName: \"kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.758983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.758832    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.758983 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.758879    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.759500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.759469    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.761516 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.761492    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.813157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.813073    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwvz\" (UniqueName: \"kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz\") pod \"maas-api-revoke-sub-ec843018-vdmhq\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:04.841912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:04.841872    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:05.208640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:05.208616    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"]
Nov 03 13:33:05.210759 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:33:05.210708    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f5636a_0641_4ca1_bebb_2c90e9d89bb4.slice/crio-e143bc23dd59e8299cb498630fa85d38c08d140b36ba4c5dc08d3a59a67b2462 WatchSource:0}: Error finding container e143bc23dd59e8299cb498630fa85d38c08d140b36ba4c5dc08d3a59a67b2462: Status 404 returned error can't find the container with id e143bc23dd59e8299cb498630fa85d38c08d140b36ba4c5dc08d3a59a67b2462
Nov 03 13:33:06.002142 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:06.002101    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq" event={"ID":"58f5636a-0641-4ca1-bebb-2c90e9d89bb4","Type":"ContainerStarted","Data":"260af890e71273be8326e848479f8230dd15b5ed947e41e20e8772d9bccc01a1"}
Nov 03 13:33:06.002557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:06.002163    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq" event={"ID":"58f5636a-0641-4ca1-bebb-2c90e9d89bb4","Type":"ContainerStarted","Data":"e143bc23dd59e8299cb498630fa85d38c08d140b36ba4c5dc08d3a59a67b2462"}
Nov 03 13:33:06.109047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:06.108991    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq" podStartSLOduration=2.108976517 podStartE2EDuration="2.108976517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:06.106898366 +0000 UTC m=+6127.678482300" watchObservedRunningTime="2026-11-03 13:33:06.108976517 +0000 UTC m=+6127.680560429"
Nov 03 13:33:07.008024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:07.007989    4546 generic.go:356] "Generic (PLEG): container finished" podID="58f5636a-0641-4ca1-bebb-2c90e9d89bb4" containerID="260af890e71273be8326e848479f8230dd15b5ed947e41e20e8772d9bccc01a1" exitCode=0
Nov 03 13:33:07.008448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:07.008071    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq" event={"ID":"58f5636a-0641-4ca1-bebb-2c90e9d89bb4","Type":"ContainerDied","Data":"260af890e71273be8326e848479f8230dd15b5ed947e41e20e8772d9bccc01a1"}
Nov 03 13:33:07.651391 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:33:07.651352    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:33:08.162099 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.162067    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:08.295318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.295226    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz\" (UniqueName: \"kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz\") pod \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") "
Nov 03 13:33:08.295318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.295287    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca\") pod \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") "
Nov 03 13:33:08.295524 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.295364    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca\") pod \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\" (UID: \"58f5636a-0641-4ca1-bebb-2c90e9d89bb4\") "
Nov 03 13:33:08.295734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.295701    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca" pod "58f5636a-0641-4ca1-bebb-2c90e9d89bb4" (UID: "58f5636a-0641-4ca1-bebb-2c90e9d89bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:33:08.297779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.297727    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca" pod "58f5636a-0641-4ca1-bebb-2c90e9d89bb4" (UID: "58f5636a-0641-4ca1-bebb-2c90e9d89bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:33:08.297877 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.297800    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz" pod "58f5636a-0641-4ca1-bebb-2c90e9d89bb4" (UID: "58f5636a-0641-4ca1-bebb-2c90e9d89bb4"). InnerVolumeSpecName "kube-api-access-fkwvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:33:08.397274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.397231    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.397274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.397268    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.397274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:08.397281    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwvz\" (UniqueName: \"kubernetes.io/projected/58f5636a-0641-4ca1-bebb-2c90e9d89bb4-kube-api-access-fkwvz\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:09.018652 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:09.018623    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"
Nov 03 13:33:09.018856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:09.018626    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq" event={"ID":"58f5636a-0641-4ca1-bebb-2c90e9d89bb4","Type":"ContainerDied","Data":"e143bc23dd59e8299cb498630fa85d38c08d140b36ba4c5dc08d3a59a67b2462"}
Nov 03 13:33:09.018856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:09.018791    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e143bc23dd59e8299cb498630fa85d38c08d140b36ba4c5dc08d3a59a67b2462"
Nov 03 13:33:29.043482 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:29.043428    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"]
Nov 03 13:33:29.091743 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:29.091707    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sw8t7"]
Nov 03 13:33:30.571795 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:30.571736    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92046aca-581f-47d8-bdaf-cc1786b9a797" path="/var/lib/kubelet/pods/92046aca-581f-47d8-bdaf-cc1786b9a797/volumes"
Nov 03 13:33:37.650683 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:33:37.650598    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:33:39.005492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.005448    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"]
Nov 03 13:33:39.006066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.006044    4546 cpu_manager.go:414] "containerMap: removing container" podUID="58f5636a-0641-4ca1-bebb-2c90e9d89bb4" containerName="revoke-keys"
Nov 03 13:33:39.006150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.006070    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f5636a-0641-4ca1-bebb-2c90e9d89bb4" containerName="revoke-keys"
Nov 03 13:33:39.006208 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.006182    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f5636a-0641-4ca1-bebb-2c90e9d89bb4" containerName="revoke-keys"
Nov 03 13:33:39.009675 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.009648    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.061233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.061194    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:33:39.061233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.061196    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:33:39.066295 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.066268    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:33:39.107332 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.107299    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:33:39.109269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.109246    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"]
Nov 03 13:33:39.126158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.126124    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.126330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.126185    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.126330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.126276    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txwm\" (UniqueName: \"kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.227218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.227177    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txwm\" (UniqueName: \"kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.227448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.227312    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.227448 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.227388    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.228058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.228033    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.229935 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.229914    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.280808 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.280709    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txwm\" (UniqueName: \"kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm\") pod \"maas-api-revoke-sub-046e1343-h8knp\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.322649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.322612    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:39.686660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:39.686634    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"]
Nov 03 13:33:39.688945 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:33:39.688903    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02691eac_fb44_49a0_b043_709d3721a18a.slice/crio-a73112ba928744869ac72b7c94e1eefaa8fa077c3b9c17bda6bc40d82008e00e WatchSource:0}: Error finding container a73112ba928744869ac72b7c94e1eefaa8fa077c3b9c17bda6bc40d82008e00e: Status 404 returned error can't find the container with id a73112ba928744869ac72b7c94e1eefaa8fa077c3b9c17bda6bc40d82008e00e
Nov 03 13:33:40.174671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:40.174635    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp" event={"ID":"02691eac-fb44-49a0-b043-709d3721a18a","Type":"ContainerStarted","Data":"2dffaeaf45bc471d12bf0b1ae687bf2d8cb8a5bdde5fdc1f7e8cc07bf0246e45"}
Nov 03 13:33:40.175115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:40.174688    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp" event={"ID":"02691eac-fb44-49a0-b043-709d3721a18a","Type":"ContainerStarted","Data":"a73112ba928744869ac72b7c94e1eefaa8fa077c3b9c17bda6bc40d82008e00e"}
Nov 03 13:33:40.327291 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:40.327186    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp" podStartSLOduration=2.327169455 podStartE2EDuration="2.327169455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:40.280448512 +0000 UTC m=+6161.852032447" watchObservedRunningTime="2026-11-03 13:33:40.327169455 +0000 UTC m=+6161.898753366"
Nov 03 13:33:41.180547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:41.180512    4546 generic.go:356] "Generic (PLEG): container finished" podID="02691eac-fb44-49a0-b043-709d3721a18a" containerID="2dffaeaf45bc471d12bf0b1ae687bf2d8cb8a5bdde5fdc1f7e8cc07bf0246e45" exitCode=0
Nov 03 13:33:41.181013 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:41.180582    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp" event={"ID":"02691eac-fb44-49a0-b043-709d3721a18a","Type":"ContainerDied","Data":"2dffaeaf45bc471d12bf0b1ae687bf2d8cb8a5bdde5fdc1f7e8cc07bf0246e45"}
Nov 03 13:33:42.327117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.327089    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:42.459781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.459676    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm\" (UniqueName: \"kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm\") pod \"02691eac-fb44-49a0-b043-709d3721a18a\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") "
Nov 03 13:33:42.459781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.459771    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca\") pod \"02691eac-fb44-49a0-b043-709d3721a18a\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") "
Nov 03 13:33:42.460030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.459920    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca\") pod \"02691eac-fb44-49a0-b043-709d3721a18a\" (UID: \"02691eac-fb44-49a0-b043-709d3721a18a\") "
Nov 03 13:33:42.460238 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.460208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca" pod "02691eac-fb44-49a0-b043-709d3721a18a" (UID: "02691eac-fb44-49a0-b043-709d3721a18a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:33:42.462246 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.462227    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm" pod "02691eac-fb44-49a0-b043-709d3721a18a" (UID: "02691eac-fb44-49a0-b043-709d3721a18a"). InnerVolumeSpecName "kube-api-access-4txwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:33:42.462323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.462226    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca" pod "02691eac-fb44-49a0-b043-709d3721a18a" (UID: "02691eac-fb44-49a0-b043-709d3721a18a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:33:42.561387 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.561348    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:42.561387 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.561384    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txwm\" (UniqueName: \"kubernetes.io/projected/02691eac-fb44-49a0-b043-709d3721a18a-kube-api-access-4txwm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:42.561387 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:42.561395    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02691eac-fb44-49a0-b043-709d3721a18a-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:43.191863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:43.191824    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp" event={"ID":"02691eac-fb44-49a0-b043-709d3721a18a","Type":"ContainerDied","Data":"a73112ba928744869ac72b7c94e1eefaa8fa077c3b9c17bda6bc40d82008e00e"}
Nov 03 13:33:43.192069 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:43.191877    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a73112ba928744869ac72b7c94e1eefaa8fa077c3b9c17bda6bc40d82008e00e"
Nov 03 13:33:43.192069 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:43.191836    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"
Nov 03 13:33:49.791493 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.791451    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"]
Nov 03 13:33:49.791922 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.791872    4546 cpu_manager.go:414] "containerMap: removing container" podUID="02691eac-fb44-49a0-b043-709d3721a18a" containerName="revoke-keys"
Nov 03 13:33:49.791922 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.791886    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02691eac-fb44-49a0-b043-709d3721a18a" containerName="revoke-keys"
Nov 03 13:33:49.792019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.791980    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="02691eac-fb44-49a0-b043-709d3721a18a" containerName="revoke-keys"
Nov 03 13:33:49.795270 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.795246    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:49.890192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.890153    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:33:49.890192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.890174    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:33:49.890192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.890174    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:33:49.892304 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.892281    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"]
Nov 03 13:33:49.937267 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.937226    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cdp\" (UniqueName: \"kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:49.937422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.937371    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:49.937422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.937397    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:49.943935 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:49.943912    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:33:50.038055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.038023    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.038055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.038062    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.038355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.038167    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cdp\" (UniqueName: \"kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.038886 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.038815    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.040604 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.040580    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.092067 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.092001    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cdp\" (UniqueName: \"kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp\") pod \"maas-api-revoke-sub-d3fba056-psbb5\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.111849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.111825    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:50.474621 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:50.474593    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"]
Nov 03 13:33:50.476964 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:33:50.476923    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14731846_2596_418e_a105_9d1d757641fd.slice/crio-4f80ed0cea5df7bf64abd267a0884d30f98ae1f0bd788f384d94a5109c84ca23 WatchSource:0}: Error finding container 4f80ed0cea5df7bf64abd267a0884d30f98ae1f0bd788f384d94a5109c84ca23: Status 404 returned error can't find the container with id 4f80ed0cea5df7bf64abd267a0884d30f98ae1f0bd788f384d94a5109c84ca23
Nov 03 13:33:51.229610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:51.229582    4546 generic.go:356] "Generic (PLEG): container finished" podID="14731846-2596-418e-a105-9d1d757641fd" containerID="a4752ad06f3cb76b9e246098cbc479c3bcbace776b358631ef6223c55c5d475a" exitCode=0
Nov 03 13:33:51.229965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:51.229654    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5" event={"ID":"14731846-2596-418e-a105-9d1d757641fd","Type":"ContainerDied","Data":"a4752ad06f3cb76b9e246098cbc479c3bcbace776b358631ef6223c55c5d475a"}
Nov 03 13:33:51.229965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:51.229693    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5" event={"ID":"14731846-2596-418e-a105-9d1d757641fd","Type":"ContainerStarted","Data":"4f80ed0cea5df7bf64abd267a0884d30f98ae1f0bd788f384d94a5109c84ca23"}
Nov 03 13:33:52.382845 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.382806    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:52.461283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.461232    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca\") pod \"14731846-2596-418e-a105-9d1d757641fd\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") "
Nov 03 13:33:52.461283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.461283    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp\" (UniqueName: \"kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp\") pod \"14731846-2596-418e-a105-9d1d757641fd\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") "
Nov 03 13:33:52.461603 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.461343    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca\") pod \"14731846-2596-418e-a105-9d1d757641fd\" (UID: \"14731846-2596-418e-a105-9d1d757641fd\") "
Nov 03 13:33:52.461843 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.461808    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca" pod "14731846-2596-418e-a105-9d1d757641fd" (UID: "14731846-2596-418e-a105-9d1d757641fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:33:52.463896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.463850    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca" pod "14731846-2596-418e-a105-9d1d757641fd" (UID: "14731846-2596-418e-a105-9d1d757641fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:33:52.464234 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.464208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp" pod "14731846-2596-418e-a105-9d1d757641fd" (UID: "14731846-2596-418e-a105-9d1d757641fd"). InnerVolumeSpecName "kube-api-access-b5cdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:33:52.562579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.562478    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14731846-2596-418e-a105-9d1d757641fd-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:52.562579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.562506    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5cdp\" (UniqueName: \"kubernetes.io/projected/14731846-2596-418e-a105-9d1d757641fd-kube-api-access-b5cdp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:52.562579 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:52.562518    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14731846-2596-418e-a105-9d1d757641fd-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:53.244775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:53.244267    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5" event={"ID":"14731846-2596-418e-a105-9d1d757641fd","Type":"ContainerDied","Data":"4f80ed0cea5df7bf64abd267a0884d30f98ae1f0bd788f384d94a5109c84ca23"}
Nov 03 13:33:53.244775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:53.244336    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f80ed0cea5df7bf64abd267a0884d30f98ae1f0bd788f384d94a5109c84ca23"
Nov 03 13:33:53.244775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:53.244418    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"
Nov 03 13:33:57.043492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:57.043449    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"]
Nov 03 13:33:57.089840 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:57.089805    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-w5jnq"]
Nov 03 13:33:58.573087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:33:58.573053    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d399602-0b3d-40e3-81f4-81704d54298a" path="/var/lib/kubelet/pods/0d399602-0b3d-40e3-81f4-81704d54298a/volumes"
Nov 03 13:34:07.651394 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:34:07.651351    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:34:22.043987 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:22.043950    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"]
Nov 03 13:34:22.090009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:22.089977    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fecd4b7-rx6br"]
Nov 03 13:34:22.577876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:22.577840    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6c2e08-70e1-4652-a523-544a355d6abd" path="/var/lib/kubelet/pods/5f6c2e08-70e1-4652-a523-544a355d6abd/volumes"
Nov 03 13:34:24.371331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.371290    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"]
Nov 03 13:34:24.371741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.371709    4546 cpu_manager.go:414] "containerMap: removing container" podUID="14731846-2596-418e-a105-9d1d757641fd" containerName="revoke-keys"
Nov 03 13:34:24.371741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.371725    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14731846-2596-418e-a105-9d1d757641fd" containerName="revoke-keys"
Nov 03 13:34:24.371855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.371843    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="14731846-2596-418e-a105-9d1d757641fd" containerName="revoke-keys"
Nov 03 13:34:24.375293 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.375276    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.386226 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.386197    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"]
Nov 03 13:34:24.413919 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.413881    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cpw\" (UniqueName: \"kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.414119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.413938    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.414119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.414074    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.425300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.425270    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:34:24.425454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.425335    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:34:24.426553 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.426533    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:34:24.426666 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.426604    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:34:24.515438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.515399    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cpw\" (UniqueName: \"kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.515650 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.515452    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.515650 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.515500    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.516244 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.516219    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.518047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.518028    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.571619 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.571587    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cpw\" (UniqueName: \"kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw\") pod \"maas-api-revoke-sub-ae594979-895kk\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:24.689149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:24.689062    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:25.052665 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:25.052636    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"]
Nov 03 13:34:25.054469 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:34:25.054433    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93e3663_a5c7_4875_bddf_49de455f6e6c.slice/crio-f0cee8863307edd076053cfa360baa2163f6783ad1a30ee7ebdf29946b91bb47 WatchSource:0}: Error finding container f0cee8863307edd076053cfa360baa2163f6783ad1a30ee7ebdf29946b91bb47: Status 404 returned error can't find the container with id f0cee8863307edd076053cfa360baa2163f6783ad1a30ee7ebdf29946b91bb47
Nov 03 13:34:25.464800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:25.464734    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk" event={"ID":"a93e3663-a5c7-4875-bddf-49de455f6e6c","Type":"ContainerStarted","Data":"35f754029e6d268285bf5f9047a3585a9b3925749dff6153ba56fc083ac3ae93"}
Nov 03 13:34:25.465314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:25.464820    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk" event={"ID":"a93e3663-a5c7-4875-bddf-49de455f6e6c","Type":"ContainerStarted","Data":"f0cee8863307edd076053cfa360baa2163f6783ad1a30ee7ebdf29946b91bb47"}
Nov 03 13:34:25.571492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:25.571375    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk" podStartSLOduration=1.571358284 podStartE2EDuration="1.571358284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:25.569633755 +0000 UTC m=+6207.141217667" watchObservedRunningTime="2026-11-03 13:34:25.571358284 +0000 UTC m=+6207.142942197"
Nov 03 13:34:26.470851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:26.470818    4546 generic.go:356] "Generic (PLEG): container finished" podID="a93e3663-a5c7-4875-bddf-49de455f6e6c" containerID="35f754029e6d268285bf5f9047a3585a9b3925749dff6153ba56fc083ac3ae93" exitCode=0
Nov 03 13:34:26.471324 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:26.470894    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk" event={"ID":"a93e3663-a5c7-4875-bddf-49de455f6e6c","Type":"ContainerDied","Data":"35f754029e6d268285bf5f9047a3585a9b3925749dff6153ba56fc083ac3ae93"}
Nov 03 13:34:27.609326 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.609302    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:27.646326 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.646292    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw\" (UniqueName: \"kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw\") pod \"a93e3663-a5c7-4875-bddf-49de455f6e6c\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") "
Nov 03 13:34:27.646494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.646475    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca\") pod \"a93e3663-a5c7-4875-bddf-49de455f6e6c\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") "
Nov 03 13:34:27.646544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.646524    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca\") pod \"a93e3663-a5c7-4875-bddf-49de455f6e6c\" (UID: \"a93e3663-a5c7-4875-bddf-49de455f6e6c\") "
Nov 03 13:34:27.646887 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.646861    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca" pod "a93e3663-a5c7-4875-bddf-49de455f6e6c" (UID: "a93e3663-a5c7-4875-bddf-49de455f6e6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:27.647049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.647027    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:27.648655 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.648634    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw" pod "a93e3663-a5c7-4875-bddf-49de455f6e6c" (UID: "a93e3663-a5c7-4875-bddf-49de455f6e6c"). InnerVolumeSpecName "kube-api-access-47cpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:27.648819 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.648801    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca" pod "a93e3663-a5c7-4875-bddf-49de455f6e6c" (UID: "a93e3663-a5c7-4875-bddf-49de455f6e6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:27.748097 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.748009    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93e3663-a5c7-4875-bddf-49de455f6e6c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:27.748097 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:27.748043    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cpw\" (UniqueName: \"kubernetes.io/projected/a93e3663-a5c7-4875-bddf-49de455f6e6c-kube-api-access-47cpw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:28.481667 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:28.481633    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"
Nov 03 13:34:28.481899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:28.481666    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk" event={"ID":"a93e3663-a5c7-4875-bddf-49de455f6e6c","Type":"ContainerDied","Data":"f0cee8863307edd076053cfa360baa2163f6783ad1a30ee7ebdf29946b91bb47"}
Nov 03 13:34:28.481899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:28.481725    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0cee8863307edd076053cfa360baa2163f6783ad1a30ee7ebdf29946b91bb47"
Nov 03 13:34:37.652267 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:34:37.652225    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:34:46.104183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.104150    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"]
Nov 03 13:34:46.104607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.104550    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a93e3663-a5c7-4875-bddf-49de455f6e6c" containerName="revoke-keys"
Nov 03 13:34:46.104607 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.104562    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93e3663-a5c7-4875-bddf-49de455f6e6c" containerName="revoke-keys"
Nov 03 13:34:46.104715 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.104651    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93e3663-a5c7-4875-bddf-49de455f6e6c" containerName="revoke-keys"
Nov 03 13:34:46.107986 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.107964    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.156384 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.156341    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:34:46.156584 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.156445    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:34:46.202864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.202830    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:34:46.203061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.202920    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:34:46.204939 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.204905    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"]
Nov 03 13:34:46.257449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.257409    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.257649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.257493    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlxv\" (UniqueName: \"kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.257649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.257569    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.358906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.358815    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlxv\" (UniqueName: \"kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.358906 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.358858    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.359217 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.358919    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.359628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.359601    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.361466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.361435    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.416523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.416490    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlxv\" (UniqueName: \"kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv\") pod \"maas-api-revoke-sub-a4aef7d7-hb9h5\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.421171 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.421151    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:46.783192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:46.783025    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"]
Nov 03 13:34:46.785954 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:34:46.785926    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d3b2e3_29be_4fa9_a459_9a8942aada6c.slice/crio-9f3711a15c88b1c5d5e2e47cdea7f0b8f180f5cfba72852efc15ef1115f323c1 WatchSource:0}: Error finding container 9f3711a15c88b1c5d5e2e47cdea7f0b8f180f5cfba72852efc15ef1115f323c1: Status 404 returned error can't find the container with id 9f3711a15c88b1c5d5e2e47cdea7f0b8f180f5cfba72852efc15ef1115f323c1
Nov 03 13:34:47.604286 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:47.604249    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5" event={"ID":"96d3b2e3-29be-4fa9-a459-9a8942aada6c","Type":"ContainerStarted","Data":"d2bd462c7a8520e142abc98d204a7ead6328a0fc6444fc2328adf31baa453f7a"}
Nov 03 13:34:47.604766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:47.604303    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5" event={"ID":"96d3b2e3-29be-4fa9-a459-9a8942aada6c","Type":"ContainerStarted","Data":"9f3711a15c88b1c5d5e2e47cdea7f0b8f180f5cfba72852efc15ef1115f323c1"}
Nov 03 13:34:47.711510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:47.711458    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5" podStartSLOduration=1.71144396 podStartE2EDuration="1.71144396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:47.710137209 +0000 UTC m=+6229.281721103" watchObservedRunningTime="2026-11-03 13:34:47.71144396 +0000 UTC m=+6229.283027872"
Nov 03 13:34:48.610739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:48.610703    4546 generic.go:356] "Generic (PLEG): container finished" podID="96d3b2e3-29be-4fa9-a459-9a8942aada6c" containerID="d2bd462c7a8520e142abc98d204a7ead6328a0fc6444fc2328adf31baa453f7a" exitCode=0
Nov 03 13:34:48.611182 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:48.610768    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5" event={"ID":"96d3b2e3-29be-4fa9-a459-9a8942aada6c","Type":"ContainerDied","Data":"d2bd462c7a8520e142abc98d204a7ead6328a0fc6444fc2328adf31baa453f7a"}
Nov 03 13:34:49.764609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.764583    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:49.892663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.892577    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca\") pod \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") "
Nov 03 13:34:49.892663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.892622    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv\" (UniqueName: \"kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv\") pod \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") "
Nov 03 13:34:49.892960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.892668    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca\") pod \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\" (UID: \"96d3b2e3-29be-4fa9-a459-9a8942aada6c\") "
Nov 03 13:34:49.893058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.893032    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca" pod "96d3b2e3-29be-4fa9-a459-9a8942aada6c" (UID: "96d3b2e3-29be-4fa9-a459-9a8942aada6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:49.895020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.894997    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv" pod "96d3b2e3-29be-4fa9-a459-9a8942aada6c" (UID: "96d3b2e3-29be-4fa9-a459-9a8942aada6c"). InnerVolumeSpecName "kube-api-access-5dlxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:49.895120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.895041    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca" pod "96d3b2e3-29be-4fa9-a459-9a8942aada6c" (UID: "96d3b2e3-29be-4fa9-a459-9a8942aada6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:49.994210 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.994174    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:49.994210 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.994206    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dlxv\" (UniqueName: \"kubernetes.io/projected/96d3b2e3-29be-4fa9-a459-9a8942aada6c-kube-api-access-5dlxv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:49.994210 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:49.994216    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d3b2e3-29be-4fa9-a459-9a8942aada6c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:50.621649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:50.621620    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"
Nov 03 13:34:50.621847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:50.621644    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5" event={"ID":"96d3b2e3-29be-4fa9-a459-9a8942aada6c","Type":"ContainerDied","Data":"9f3711a15c88b1c5d5e2e47cdea7f0b8f180f5cfba72852efc15ef1115f323c1"}
Nov 03 13:34:50.621847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:34:50.621692    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3711a15c88b1c5d5e2e47cdea7f0b8f180f5cfba72852efc15ef1115f323c1"
Nov 03 13:35:03.235875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:03.235844    4546 scope.go:122] "RemoveContainer" containerID="8b6c30c296f060aca31268abde64bbd3f760af9a4043ce782a516fb1bf53df1e"
Nov 03 13:35:03.244827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:03.244801    4546 scope.go:122] "RemoveContainer" containerID="c5bd0e56df46fb0334bdee510eed785a49f18fa2183e16aef1f0bf273b246bda"
Nov 03 13:35:07.391044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.391007    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"]
Nov 03 13:35:07.391618 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.391588    4546 cpu_manager.go:414] "containerMap: removing container" podUID="96d3b2e3-29be-4fa9-a459-9a8942aada6c" containerName="revoke-keys"
Nov 03 13:35:07.391618 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.391611    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d3b2e3-29be-4fa9-a459-9a8942aada6c" containerName="revoke-keys"
Nov 03 13:35:07.391929 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.391678    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d3b2e3-29be-4fa9-a459-9a8942aada6c" containerName="revoke-keys"
Nov 03 13:35:07.395891 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.395868    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.447167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.447091    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:07.485673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.485292    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.485673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.485382    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.485673 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.485462    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5bb\" (UniqueName: \"kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.498091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.497912    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:07.498091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.497932    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:07.498091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.498081    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:35:07.499662 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.499638    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"]
Nov 03 13:35:07.589853 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.589737    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5bb\" (UniqueName: \"kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.590348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.590317    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.590488 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.590386    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.591517 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.591491    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.593463 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.593384    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.687879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.687793    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5bb\" (UniqueName: \"kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb\") pod \"maas-api-revoke-sub-79805ca9-r48jj\" (UID: \"30497586-d966-4352-820e-aae658820eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.733817 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:07.722412    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:07.733817 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:35:07.730928    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:35:08.228849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:08.222648    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"]
Nov 03 13:35:08.754901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:08.754860    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj" event={"ID":"30497586-d966-4352-820e-aae658820eee","Type":"ContainerStarted","Data":"080f567e58b9ffe7690cea2e712d87224ebb8a0496e504b45c912b3ce41adc44"}
Nov 03 13:35:08.755352 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:08.754925    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj" event={"ID":"30497586-d966-4352-820e-aae658820eee","Type":"ContainerStarted","Data":"86de44272943353fbcfd42be508824b93342f897b0d1712001bc85f21e9707b3"}
Nov 03 13:35:08.863114 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:08.863000    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj" podStartSLOduration=1.8629845550000002 podStartE2EDuration="1.862984555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:08.860959323 +0000 UTC m=+6250.432543245" watchObservedRunningTime="2026-11-03 13:35:08.862984555 +0000 UTC m=+6250.434568490"
Nov 03 13:35:09.760859 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:09.760826    4546 generic.go:356] "Generic (PLEG): container finished" podID="30497586-d966-4352-820e-aae658820eee" containerID="080f567e58b9ffe7690cea2e712d87224ebb8a0496e504b45c912b3ce41adc44" exitCode=0
Nov 03 13:35:09.761310 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:09.760912    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj" event={"ID":"30497586-d966-4352-820e-aae658820eee","Type":"ContainerDied","Data":"080f567e58b9ffe7690cea2e712d87224ebb8a0496e504b45c912b3ce41adc44"}
Nov 03 13:35:10.901245 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.901218    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:10.923323 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.923297    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca\") pod \"30497586-d966-4352-820e-aae658820eee\" (UID: \"30497586-d966-4352-820e-aae658820eee\") "
Nov 03 13:35:10.923420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.923383    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb\" (UniqueName: \"kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb\") pod \"30497586-d966-4352-820e-aae658820eee\" (UID: \"30497586-d966-4352-820e-aae658820eee\") "
Nov 03 13:35:10.923473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.923456    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca\") pod \"30497586-d966-4352-820e-aae658820eee\" (UID: \"30497586-d966-4352-820e-aae658820eee\") "
Nov 03 13:35:10.923823 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.923798    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca" pod "30497586-d966-4352-820e-aae658820eee" (UID: "30497586-d966-4352-820e-aae658820eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:10.925517 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.925485    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca" pod "30497586-d966-4352-820e-aae658820eee" (UID: "30497586-d966-4352-820e-aae658820eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:10.925631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:10.925567    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb" pod "30497586-d966-4352-820e-aae658820eee" (UID: "30497586-d966-4352-820e-aae658820eee"). InnerVolumeSpecName "kube-api-access-4c5bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:11.024542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:11.024463    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5bb\" (UniqueName: \"kubernetes.io/projected/30497586-d966-4352-820e-aae658820eee-kube-api-access-4c5bb\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:11.024542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:11.024490    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30497586-d966-4352-820e-aae658820eee-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:11.024542 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:11.024501    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30497586-d966-4352-820e-aae658820eee-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:11.771449 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:11.771421    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"
Nov 03 13:35:11.771663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:11.771453    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj" event={"ID":"30497586-d966-4352-820e-aae658820eee","Type":"ContainerDied","Data":"86de44272943353fbcfd42be508824b93342f897b0d1712001bc85f21e9707b3"}
Nov 03 13:35:11.771663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:11.771500    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86de44272943353fbcfd42be508824b93342f897b0d1712001bc85f21e9707b3"
Nov 03 13:35:24.808855 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.808811    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"]
Nov 03 13:35:24.811321 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.809413    4546 cpu_manager.go:414] "containerMap: removing container" podUID="30497586-d966-4352-820e-aae658820eee" containerName="revoke-keys"
Nov 03 13:35:24.811321 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.809435    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30497586-d966-4352-820e-aae658820eee" containerName="revoke-keys"
Nov 03 13:35:24.811321 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.809585    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="30497586-d966-4352-820e-aae658820eee" containerName="revoke-keys"
Nov 03 13:35:24.813143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.813120    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.863672 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.863627    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:24.868291 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.868252    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.868563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.868543    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.868782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.868734    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkwf\" (UniqueName: \"kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.909793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.909670    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:24.909793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.909708    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:24.909793 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.909729    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:35:24.915969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.915933    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"]
Nov 03 13:35:24.969790 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.969734    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.970074 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.970052    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.970188 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.970169    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkwf\" (UniqueName: \"kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.970649 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.970621    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:24.973072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:24.973041    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:25.024139 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:25.024100    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkwf\" (UniqueName: \"kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf\") pod \"maas-api-revoke-sub-0b7e8edb-22qzs\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:25.129626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:25.129514    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:25.553478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:25.553434    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"]
Nov 03 13:35:25.861829 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:25.861203    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs" event={"ID":"18400f5e-213f-4ee3-9981-d17ab5820b7d","Type":"ContainerStarted","Data":"be2e53e7154fb8d4ae9dc09d18e1e8afbccf91b8a5e8d409c7011111773fb733"}
Nov 03 13:35:26.878986 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:26.878328    4546 generic.go:356] "Generic (PLEG): container finished" podID="18400f5e-213f-4ee3-9981-d17ab5820b7d" containerID="4fbaa32d6ef6b7529904148e9d6e30731eae3bdd1c321c005a1fef2c6ace2bd7" exitCode=0
Nov 03 13:35:26.878986 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:26.878498    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs" event={"ID":"18400f5e-213f-4ee3-9981-d17ab5820b7d","Type":"ContainerDied","Data":"4fbaa32d6ef6b7529904148e9d6e30731eae3bdd1c321c005a1fef2c6ace2bd7"}
Nov 03 13:35:28.018725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.018700    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:28.114705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.114673    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf\" (UniqueName: \"kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf\") pod \"18400f5e-213f-4ee3-9981-d17ab5820b7d\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") "
Nov 03 13:35:28.114975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.114805    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca\") pod \"18400f5e-213f-4ee3-9981-d17ab5820b7d\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") "
Nov 03 13:35:28.114975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.114852    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca\") pod \"18400f5e-213f-4ee3-9981-d17ab5820b7d\" (UID: \"18400f5e-213f-4ee3-9981-d17ab5820b7d\") "
Nov 03 13:35:28.115225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.115197    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca" pod "18400f5e-213f-4ee3-9981-d17ab5820b7d" (UID: "18400f5e-213f-4ee3-9981-d17ab5820b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:28.117221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.117194    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca" pod "18400f5e-213f-4ee3-9981-d17ab5820b7d" (UID: "18400f5e-213f-4ee3-9981-d17ab5820b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:28.117327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.117220    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf" pod "18400f5e-213f-4ee3-9981-d17ab5820b7d" (UID: "18400f5e-213f-4ee3-9981-d17ab5820b7d"). InnerVolumeSpecName "kube-api-access-xbkwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:28.215971 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.215942    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:28.216136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.215974    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18400f5e-213f-4ee3-9981-d17ab5820b7d-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:28.216136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.215985    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkwf\" (UniqueName: \"kubernetes.io/projected/18400f5e-213f-4ee3-9981-d17ab5820b7d-kube-api-access-xbkwf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:28.889458 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.889359    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"
Nov 03 13:35:28.889458 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.889384    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs" event={"ID":"18400f5e-213f-4ee3-9981-d17ab5820b7d","Type":"ContainerDied","Data":"be2e53e7154fb8d4ae9dc09d18e1e8afbccf91b8a5e8d409c7011111773fb733"}
Nov 03 13:35:28.889458 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:28.889446    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2e53e7154fb8d4ae9dc09d18e1e8afbccf91b8a5e8d409c7011111773fb733"
Nov 03 13:35:36.050898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.050862    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"]
Nov 03 13:35:36.051430 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.051411    4546 cpu_manager.go:414] "containerMap: removing container" podUID="18400f5e-213f-4ee3-9981-d17ab5820b7d" containerName="revoke-keys"
Nov 03 13:35:36.051492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.051432    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18400f5e-213f-4ee3-9981-d17ab5820b7d" containerName="revoke-keys"
Nov 03 13:35:36.051539 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.051507    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="18400f5e-213f-4ee3-9981-d17ab5820b7d" containerName="revoke-keys"
Nov 03 13:35:36.054739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.054719    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.103221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.103195    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:36.149437 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.149402    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:36.149437 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.149405    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:36.149681 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.149482    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:35:36.151774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.151723    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"]
Nov 03 13:35:36.198452 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.198414    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.198642 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.198462    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lw4\" (UniqueName: \"kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.198642 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.198619    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.299803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.299743    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.300034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.299819    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lw4\" (UniqueName: \"kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.300034 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.299928    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.300545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.300507    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.302483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.302427    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.352997 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.352964    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lw4\" (UniqueName: \"kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4\") pod \"maas-api-revoke-sub-3d4156f1-cwhbr\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.367376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.367348    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:36.732533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.732509    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"]
Nov 03 13:35:36.734595 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:35:36.734567    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45c585d_7c6f_4f0a_baa3_766b992b337c.slice/crio-ed5d8dac9bd5f7197d2256080f54524679311f2cbbd64df212a79f3ab838c2a6 WatchSource:0}: Error finding container ed5d8dac9bd5f7197d2256080f54524679311f2cbbd64df212a79f3ab838c2a6: Status 404 returned error can't find the container with id ed5d8dac9bd5f7197d2256080f54524679311f2cbbd64df212a79f3ab838c2a6
Nov 03 13:35:36.928994 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:36.928957    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr" event={"ID":"c45c585d-7c6f-4f0a-baa3-766b992b337c","Type":"ContainerStarted","Data":"ed5d8dac9bd5f7197d2256080f54524679311f2cbbd64df212a79f3ab838c2a6"}
Nov 03 13:35:37.650326 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:35:37.650292    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:35:37.935478 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:37.935381    4546 generic.go:356] "Generic (PLEG): container finished" podID="c45c585d-7c6f-4f0a-baa3-766b992b337c" containerID="4c05847a2eced35ba04da63ea751df8c0c8a5c7fe717d6d537e6f29874c3f287" exitCode=0
Nov 03 13:35:37.935676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:37.935466    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr" event={"ID":"c45c585d-7c6f-4f0a-baa3-766b992b337c","Type":"ContainerDied","Data":"4c05847a2eced35ba04da63ea751df8c0c8a5c7fe717d6d537e6f29874c3f287"}
Nov 03 13:35:39.074639 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.074609    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:39.232724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.232691    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca\") pod \"c45c585d-7c6f-4f0a-baa3-766b992b337c\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") "
Nov 03 13:35:39.232930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.232825    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca\") pod \"c45c585d-7c6f-4f0a-baa3-766b992b337c\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") "
Nov 03 13:35:39.232930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.232886    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4\" (UniqueName: \"kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4\") pod \"c45c585d-7c6f-4f0a-baa3-766b992b337c\" (UID: \"c45c585d-7c6f-4f0a-baa3-766b992b337c\") "
Nov 03 13:35:39.233167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.233139    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca" pod "c45c585d-7c6f-4f0a-baa3-766b992b337c" (UID: "c45c585d-7c6f-4f0a-baa3-766b992b337c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:39.235151 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.235122    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca" pod "c45c585d-7c6f-4f0a-baa3-766b992b337c" (UID: "c45c585d-7c6f-4f0a-baa3-766b992b337c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:39.235266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.235183    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4" pod "c45c585d-7c6f-4f0a-baa3-766b992b337c" (UID: "c45c585d-7c6f-4f0a-baa3-766b992b337c"). InnerVolumeSpecName "kube-api-access-49lw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:39.334583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.334543    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49lw4\" (UniqueName: \"kubernetes.io/projected/c45c585d-7c6f-4f0a-baa3-766b992b337c-kube-api-access-49lw4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:39.334583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.334576    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:39.334583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.334587    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45c585d-7c6f-4f0a-baa3-766b992b337c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:39.946059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.946028    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"
Nov 03 13:35:39.946059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.946044    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr" event={"ID":"c45c585d-7c6f-4f0a-baa3-766b992b337c","Type":"ContainerDied","Data":"ed5d8dac9bd5f7197d2256080f54524679311f2cbbd64df212a79f3ab838c2a6"}
Nov 03 13:35:39.946395 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:39.946099    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5d8dac9bd5f7197d2256080f54524679311f2cbbd64df212a79f3ab838c2a6"
Nov 03 13:35:59.924156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:59.924107    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"]
Nov 03 13:35:59.970164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:35:59.970109    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwhbr"]
Nov 03 13:36:00.046771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.046724    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"]
Nov 03 13:36:00.047215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.047201    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c45c585d-7c6f-4f0a-baa3-766b992b337c" containerName="revoke-keys"
Nov 03 13:36:00.047261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.047217    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45c585d-7c6f-4f0a-baa3-766b992b337c" containerName="revoke-keys"
Nov 03 13:36:00.047305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.047296    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45c585d-7c6f-4f0a-baa3-766b992b337c" containerName="revoke-keys"
Nov 03 13:36:00.050434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.050415    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.098581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.098542    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.098816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.098596    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrmq\" (UniqueName: \"kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.098816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.098734    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.102277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.102252    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:36:00.102456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.102321    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:36:00.102456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.102378    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:36:00.102456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.102416    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:36:00.104947 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.104914    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"]
Nov 03 13:36:00.199816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.199720    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.199816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.199802    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrmq\" (UniqueName: \"kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.200091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.199862    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.200414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.200389    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.202437 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.202421    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.255104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.255064    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrmq\" (UniqueName: \"kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq\") pod \"maas-api-revoke-sub-3d4156f1-khgnm\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.362836 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.362796    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:00.571482 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.571446    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45c585d-7c6f-4f0a-baa3-766b992b337c" path="/var/lib/kubelet/pods/c45c585d-7c6f-4f0a-baa3-766b992b337c/volumes"
Nov 03 13:36:00.727527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:00.727501    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"]
Nov 03 13:36:00.730017 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:36:00.729969    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc918c20_e4c8_4ff1_805b_6c2f194554f0.slice/crio-e89ae7a98642ed90269e838d7d2c2fe749838ea931393898d9304f1f92c840e0 WatchSource:0}: Error finding container e89ae7a98642ed90269e838d7d2c2fe749838ea931393898d9304f1f92c840e0: Status 404 returned error can't find the container with id e89ae7a98642ed90269e838d7d2c2fe749838ea931393898d9304f1f92c840e0
Nov 03 13:36:01.098180 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:01.098142    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm" event={"ID":"bc918c20-e4c8-4ff1-805b-6c2f194554f0","Type":"ContainerStarted","Data":"eeea41580113a80abd8a4648feeab8992388eeb398c740003fc75d3d57226823"}
Nov 03 13:36:01.098637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:01.098200    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm" event={"ID":"bc918c20-e4c8-4ff1-805b-6c2f194554f0","Type":"ContainerStarted","Data":"e89ae7a98642ed90269e838d7d2c2fe749838ea931393898d9304f1f92c840e0"}
Nov 03 13:36:01.208895 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:01.208790    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm" podStartSLOduration=1.208774564 podStartE2EDuration="1.208774564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:01.206054843 +0000 UTC m=+6302.777638755" watchObservedRunningTime="2026-11-03 13:36:01.208774564 +0000 UTC m=+6302.780358509"
Nov 03 13:36:02.104496 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:02.104462    4546 generic.go:356] "Generic (PLEG): container finished" podID="bc918c20-e4c8-4ff1-805b-6c2f194554f0" containerID="eeea41580113a80abd8a4648feeab8992388eeb398c740003fc75d3d57226823" exitCode=0
Nov 03 13:36:02.104956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:02.104527    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm" event={"ID":"bc918c20-e4c8-4ff1-805b-6c2f194554f0","Type":"ContainerDied","Data":"eeea41580113a80abd8a4648feeab8992388eeb398c740003fc75d3d57226823"}
Nov 03 13:36:03.248731 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.248697    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:03.269417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.269387    4546 scope.go:122] "RemoveContainer" containerID="d4e59782e9d05e267638a2f2e24e5d7b0a72abda1a105cea27be2ead914b50b3"
Nov 03 13:36:03.330764 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.330718    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq\" (UniqueName: \"kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq\") pod \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") "
Nov 03 13:36:03.330962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.330852    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca\") pod \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") "
Nov 03 13:36:03.330962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.330893    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca\") pod \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\" (UID: \"bc918c20-e4c8-4ff1-805b-6c2f194554f0\") "
Nov 03 13:36:03.331260 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.331238    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca" pod "bc918c20-e4c8-4ff1-805b-6c2f194554f0" (UID: "bc918c20-e4c8-4ff1-805b-6c2f194554f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:03.333227 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.333203    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca" pod "bc918c20-e4c8-4ff1-805b-6c2f194554f0" (UID: "bc918c20-e4c8-4ff1-805b-6c2f194554f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:36:03.333331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.333226    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq" pod "bc918c20-e4c8-4ff1-805b-6c2f194554f0" (UID: "bc918c20-e4c8-4ff1-805b-6c2f194554f0"). InnerVolumeSpecName "kube-api-access-vlrmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:03.432669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.432579    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrmq\" (UniqueName: \"kubernetes.io/projected/bc918c20-e4c8-4ff1-805b-6c2f194554f0-kube-api-access-vlrmq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:03.432669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.432614    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:03.432669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:03.432623    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc918c20-e4c8-4ff1-805b-6c2f194554f0-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:04.116056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:04.116023    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"
Nov 03 13:36:04.116250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:04.116057    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm" event={"ID":"bc918c20-e4c8-4ff1-805b-6c2f194554f0","Type":"ContainerDied","Data":"e89ae7a98642ed90269e838d7d2c2fe749838ea931393898d9304f1f92c840e0"}
Nov 03 13:36:04.116250 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:04.116103    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89ae7a98642ed90269e838d7d2c2fe749838ea931393898d9304f1f92c840e0"
Nov 03 13:36:07.650603 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:36:07.650561    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:36:37.651043 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:36:37.650958    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:36:45.154044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.154008    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"]
Nov 03 13:36:45.154591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.154409    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bc918c20-e4c8-4ff1-805b-6c2f194554f0" containerName="revoke-keys"
Nov 03 13:36:45.154591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.154421    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc918c20-e4c8-4ff1-805b-6c2f194554f0" containerName="revoke-keys"
Nov 03 13:36:45.154591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.154516    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc918c20-e4c8-4ff1-805b-6c2f194554f0" containerName="revoke-keys"
Nov 03 13:36:45.157881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.157861    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.206687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.206657    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:36:45.209277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.209256    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:36:45.255561 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.255522    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:36:45.255812 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.255567    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:36:45.256578 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.256550    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"]
Nov 03 13:36:45.291052 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.291021    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpqq\" (UniqueName: \"kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.291220 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.291158    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.291220 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.291217    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.392532 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.392497    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.392532 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.392541    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.392802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.392603    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpqq\" (UniqueName: \"kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.393314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.393294    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.395381 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.395360    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.446466 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.446391    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpqq\" (UniqueName: \"kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq\") pod \"maas-api-revoke-sub-e2fc9c5b-2cx8d\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.472075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.472041    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:45.836529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:45.836503    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"]
Nov 03 13:36:45.838957 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:36:45.838920    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d45e8b_e78d_4f5b_9c04_b476921947b0.slice/crio-27f3ac8266f6c21395060c1d0ad85264f072c321a933af08d5929597e1ce0e79 WatchSource:0}: Error finding container 27f3ac8266f6c21395060c1d0ad85264f072c321a933af08d5929597e1ce0e79: Status 404 returned error can't find the container with id 27f3ac8266f6c21395060c1d0ad85264f072c321a933af08d5929597e1ce0e79
Nov 03 13:36:46.344996 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:46.344951    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d" event={"ID":"d6d45e8b-e78d-4f5b-9c04-b476921947b0","Type":"ContainerStarted","Data":"b86bc6a40bd0c998c227722f8da47855ab560621906ad8210095740d63fdf362"}
Nov 03 13:36:46.345420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:46.345020    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d" event={"ID":"d6d45e8b-e78d-4f5b-9c04-b476921947b0","Type":"ContainerStarted","Data":"27f3ac8266f6c21395060c1d0ad85264f072c321a933af08d5929597e1ce0e79"}
Nov 03 13:36:46.453930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:46.453813    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d" podStartSLOduration=1.453796755 podStartE2EDuration="1.453796755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:46.451103086 +0000 UTC m=+6348.022687037" watchObservedRunningTime="2026-11-03 13:36:46.453796755 +0000 UTC m=+6348.025380670"
Nov 03 13:36:47.352589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:47.352557    4546 generic.go:356] "Generic (PLEG): container finished" podID="d6d45e8b-e78d-4f5b-9c04-b476921947b0" containerID="b86bc6a40bd0c998c227722f8da47855ab560621906ad8210095740d63fdf362" exitCode=0
Nov 03 13:36:47.353036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:47.352647    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d" event={"ID":"d6d45e8b-e78d-4f5b-9c04-b476921947b0","Type":"ContainerDied","Data":"b86bc6a40bd0c998c227722f8da47855ab560621906ad8210095740d63fdf362"}
Nov 03 13:36:48.500194 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.500166    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:48.623648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.623556    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq\" (UniqueName: \"kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq\") pod \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") "
Nov 03 13:36:48.623648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.623596    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca\") pod \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") "
Nov 03 13:36:48.623920 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.623719    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca\") pod \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\" (UID: \"d6d45e8b-e78d-4f5b-9c04-b476921947b0\") "
Nov 03 13:36:48.624052 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.624026    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca" pod "d6d45e8b-e78d-4f5b-9c04-b476921947b0" (UID: "d6d45e8b-e78d-4f5b-9c04-b476921947b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:48.625962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.625938    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca" pod "d6d45e8b-e78d-4f5b-9c04-b476921947b0" (UID: "d6d45e8b-e78d-4f5b-9c04-b476921947b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:36:48.626096 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.626036    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq" pod "d6d45e8b-e78d-4f5b-9c04-b476921947b0" (UID: "d6d45e8b-e78d-4f5b-9c04-b476921947b0"). InnerVolumeSpecName "kube-api-access-6cpqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:48.725348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.725312    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpqq\" (UniqueName: \"kubernetes.io/projected/d6d45e8b-e78d-4f5b-9c04-b476921947b0-kube-api-access-6cpqq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.725348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.725343    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.725348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:48.725353    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d45e8b-e78d-4f5b-9c04-b476921947b0-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:49.366382 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:49.366354    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"
Nov 03 13:36:49.366571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:49.366392    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d" event={"ID":"d6d45e8b-e78d-4f5b-9c04-b476921947b0","Type":"ContainerDied","Data":"27f3ac8266f6c21395060c1d0ad85264f072c321a933af08d5929597e1ce0e79"}
Nov 03 13:36:49.366571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:49.366440    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27f3ac8266f6c21395060c1d0ad85264f072c321a933af08d5929597e1ce0e79"
Nov 03 13:36:55.900410 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.900375    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"]
Nov 03 13:36:55.900856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.900788    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d6d45e8b-e78d-4f5b-9c04-b476921947b0" containerName="revoke-keys"
Nov 03 13:36:55.900856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.900800    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d45e8b-e78d-4f5b-9c04-b476921947b0" containerName="revoke-keys"
Nov 03 13:36:55.900970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.900890    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d45e8b-e78d-4f5b-9c04-b476921947b0" containerName="revoke-keys"
Nov 03 13:36:55.904028 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.904009    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:55.952888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.952854    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:36:55.952888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.952865    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:36:55.953178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.952908    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:36:55.998157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.998126    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b784f\" (UniqueName: \"kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:55.998331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.998165    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:55.998331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.998261    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:55.999286 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:55.999263    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:36:56.001567 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.001543    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"]
Nov 03 13:36:56.099304 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.099268    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b784f\" (UniqueName: \"kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.099304 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.099305    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.099535 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.099335    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.100059 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.100035    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.101810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.101785    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.153133 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.153059    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b784f\" (UniqueName: \"kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f\") pod \"maas-api-revoke-sub-5a29efb3-sd7gk\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.218235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.218203    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:56.580834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:56.580810    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"]
Nov 03 13:36:56.582792 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:36:56.582768    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08684a72_b72d_42b0_bcb2_5131fed86624.slice/crio-560eb66c964f6f54a5070ebdc001e89058d7ca97038d0ed92b2af556c55b2bea WatchSource:0}: Error finding container 560eb66c964f6f54a5070ebdc001e89058d7ca97038d0ed92b2af556c55b2bea: Status 404 returned error can't find the container with id 560eb66c964f6f54a5070ebdc001e89058d7ca97038d0ed92b2af556c55b2bea
Nov 03 13:36:57.409212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:57.409172    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk" event={"ID":"08684a72-b72d-42b0-bcb2-5131fed86624","Type":"ContainerStarted","Data":"a54f35c114f6ae2237eebace58b9b0898f61d00e09e5bc1343eabf0bd8a76546"}
Nov 03 13:36:57.409601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:57.409231    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk" event={"ID":"08684a72-b72d-42b0-bcb2-5131fed86624","Type":"ContainerStarted","Data":"560eb66c964f6f54a5070ebdc001e89058d7ca97038d0ed92b2af556c55b2bea"}
Nov 03 13:36:57.518653 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:57.518605    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk" podStartSLOduration=2.518590622 podStartE2EDuration="2.518590622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:57.516317345 +0000 UTC m=+6359.087901260" watchObservedRunningTime="2026-11-03 13:36:57.518590622 +0000 UTC m=+6359.090174534"
Nov 03 13:36:58.417503 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:58.417460    4546 generic.go:356] "Generic (PLEG): container finished" podID="08684a72-b72d-42b0-bcb2-5131fed86624" containerID="a54f35c114f6ae2237eebace58b9b0898f61d00e09e5bc1343eabf0bd8a76546" exitCode=0
Nov 03 13:36:58.418003 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:58.417541    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk" event={"ID":"08684a72-b72d-42b0-bcb2-5131fed86624","Type":"ContainerDied","Data":"a54f35c114f6ae2237eebace58b9b0898f61d00e09e5bc1343eabf0bd8a76546"}
Nov 03 13:36:59.888068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.888026    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:36:59.988902 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.988852    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca\") pod \"08684a72-b72d-42b0-bcb2-5131fed86624\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") "
Nov 03 13:36:59.989143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.988921    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f\" (UniqueName: \"kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f\") pod \"08684a72-b72d-42b0-bcb2-5131fed86624\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") "
Nov 03 13:36:59.989143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.989042    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca\") pod \"08684a72-b72d-42b0-bcb2-5131fed86624\" (UID: \"08684a72-b72d-42b0-bcb2-5131fed86624\") "
Nov 03 13:36:59.989436 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.989409    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca" pod "08684a72-b72d-42b0-bcb2-5131fed86624" (UID: "08684a72-b72d-42b0-bcb2-5131fed86624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:59.991172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.991146    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f" pod "08684a72-b72d-42b0-bcb2-5131fed86624" (UID: "08684a72-b72d-42b0-bcb2-5131fed86624"). InnerVolumeSpecName "kube-api-access-b784f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:59.991314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:36:59.991295    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca" pod "08684a72-b72d-42b0-bcb2-5131fed86624" (UID: "08684a72-b72d-42b0-bcb2-5131fed86624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:37:00.097899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.092470    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:00.097899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.092508    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b784f\" (UniqueName: \"kubernetes.io/projected/08684a72-b72d-42b0-bcb2-5131fed86624-kube-api-access-b784f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:00.097899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.092524    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08684a72-b72d-42b0-bcb2-5131fed86624-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:00.097899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.093018    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"]
Nov 03 13:37:00.097899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.096476    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-brlzr"]
Nov 03 13:37:00.470273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.470228    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk" event={"ID":"08684a72-b72d-42b0-bcb2-5131fed86624","Type":"ContainerDied","Data":"560eb66c964f6f54a5070ebdc001e89058d7ca97038d0ed92b2af556c55b2bea"}
Nov 03 13:37:00.470640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.470613    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="560eb66c964f6f54a5070ebdc001e89058d7ca97038d0ed92b2af556c55b2bea"
Nov 03 13:37:00.470901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.470883    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"
Nov 03 13:37:00.478944 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.478913    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:37:00.479198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.479172    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:37:00.595884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:00.595856    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c" path="/var/lib/kubelet/pods/a9eeb7c1-440e-40bb-9a1d-64f80ba4d29c/volumes"
Nov 03 13:37:07.650091 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:37:07.650055    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:37:17.880702 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:17.880644    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"]
Nov 03 13:37:17.926672 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:17.926632    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-khgnm"]
Nov 03 13:37:18.000796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.000762    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"]
Nov 03 13:37:18.001201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.001187    4546 cpu_manager.go:414] "containerMap: removing container" podUID="08684a72-b72d-42b0-bcb2-5131fed86624" containerName="revoke-keys"
Nov 03 13:37:18.001257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.001204    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08684a72-b72d-42b0-bcb2-5131fed86624" containerName="revoke-keys"
Nov 03 13:37:18.001302 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.001283    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="08684a72-b72d-42b0-bcb2-5131fed86624" containerName="revoke-keys"
Nov 03 13:37:18.005785 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.005738    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.056223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.056192    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:37:18.056223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.056216    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:37:18.056490 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.056219    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:37:18.056490 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.056362    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:37:18.058877 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.058848    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"]
Nov 03 13:37:18.066926 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.066899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.067091 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.066944    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bflr\" (UniqueName: \"kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.067144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.067102    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.167986 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.167891    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bflr\" (UniqueName: \"kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.168189 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.167998    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.168189 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.168046    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.168880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.168685    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.170623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.170595    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.222630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.222586    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bflr\" (UniqueName: \"kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr\") pod \"maas-api-revoke-sub-3d4156f1-rvgdg\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.319042 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.319003    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:18.579251 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.579214    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc918c20-e4c8-4ff1-805b-6c2f194554f0" path="/var/lib/kubelet/pods/bc918c20-e4c8-4ff1-805b-6c2f194554f0/volumes"
Nov 03 13:37:18.687816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:18.687789    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"]
Nov 03 13:37:19.567280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:19.567246    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg" event={"ID":"34b801e1-162e-4b42-b092-ef567a5ba407","Type":"ContainerStarted","Data":"b82f1d8d781783ce442638a835ec5787cb5749a4f8df373e3d5dd3f600585e5f"}
Nov 03 13:37:19.567717 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:19.567304    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg" event={"ID":"34b801e1-162e-4b42-b092-ef567a5ba407","Type":"ContainerStarted","Data":"fcab69ab6b9ca81077b9853824da0b66ab7658762865390964002ec8b401065d"}
Nov 03 13:37:19.722462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:19.722401    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg" podStartSLOduration=2.7223850450000002 podStartE2EDuration="2.722385045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:19.674525243 +0000 UTC m=+6381.246109155" watchObservedRunningTime="2026-11-03 13:37:19.722385045 +0000 UTC m=+6381.293968956"
Nov 03 13:37:20.573248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:20.573213    4546 generic.go:356] "Generic (PLEG): container finished" podID="34b801e1-162e-4b42-b092-ef567a5ba407" containerID="b82f1d8d781783ce442638a835ec5787cb5749a4f8df373e3d5dd3f600585e5f" exitCode=0
Nov 03 13:37:20.573725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:20.573264    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg" event={"ID":"34b801e1-162e-4b42-b092-ef567a5ba407","Type":"ContainerDied","Data":"b82f1d8d781783ce442638a835ec5787cb5749a4f8df373e3d5dd3f600585e5f"}
Nov 03 13:37:21.711385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.711357    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:21.805550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.805512    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr\" (UniqueName: \"kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr\") pod \"34b801e1-162e-4b42-b092-ef567a5ba407\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") "
Nov 03 13:37:21.805803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.805627    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca\") pod \"34b801e1-162e-4b42-b092-ef567a5ba407\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") "
Nov 03 13:37:21.805803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.805696    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca\") pod \"34b801e1-162e-4b42-b092-ef567a5ba407\" (UID: \"34b801e1-162e-4b42-b092-ef567a5ba407\") "
Nov 03 13:37:21.806093 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.806065    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca" pod "34b801e1-162e-4b42-b092-ef567a5ba407" (UID: "34b801e1-162e-4b42-b092-ef567a5ba407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:37:21.807938 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.807915    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca" pod "34b801e1-162e-4b42-b092-ef567a5ba407" (UID: "34b801e1-162e-4b42-b092-ef567a5ba407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:37:21.808053 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.808034    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr" pod "34b801e1-162e-4b42-b092-ef567a5ba407" (UID: "34b801e1-162e-4b42-b092-ef567a5ba407"). InnerVolumeSpecName "kube-api-access-8bflr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:37:21.907498 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.907410    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:21.907498 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.907441    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b801e1-162e-4b42-b092-ef567a5ba407-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:21.907498 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:21.907450    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bflr\" (UniqueName: \"kubernetes.io/projected/34b801e1-162e-4b42-b092-ef567a5ba407-kube-api-access-8bflr\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:22.584125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:22.584101    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"
Nov 03 13:37:22.584297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:22.584140    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg" event={"ID":"34b801e1-162e-4b42-b092-ef567a5ba407","Type":"ContainerDied","Data":"fcab69ab6b9ca81077b9853824da0b66ab7658762865390964002ec8b401065d"}
Nov 03 13:37:22.584297 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:22.584186    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcab69ab6b9ca81077b9853824da0b66ab7658762865390964002ec8b401065d"
Nov 03 13:37:31.050814 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:31.050779    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"]
Nov 03 13:37:31.096674 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:31.096630    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xk2l6"]
Nov 03 13:37:32.571277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:32.571229    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c141f38-a6c2-49c7-bf8f-70cce0d3eaae" path="/var/lib/kubelet/pods/8c141f38-a6c2-49c7-bf8f-70cce0d3eaae/volumes"
Nov 03 13:37:34.676093 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.676060    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"]
Nov 03 13:37:34.677348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.677319    4546 cpu_manager.go:414] "containerMap: removing container" podUID="34b801e1-162e-4b42-b092-ef567a5ba407" containerName="revoke-keys"
Nov 03 13:37:34.677547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.677530    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b801e1-162e-4b42-b092-ef567a5ba407" containerName="revoke-keys"
Nov 03 13:37:34.677781 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.677742    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b801e1-162e-4b42-b092-ef567a5ba407" containerName="revoke-keys"
Nov 03 13:37:34.681865 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.681844    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.736775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.736181    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a6706a-openshift-default-dockercfg-7whkg\""
Nov 03 13:37:34.760301 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.760249    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.762866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.762824    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.763117 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.763093    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.763257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.763240    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.763384 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.763369    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.783774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.763517    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.784257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.784218    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.784444 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.784424    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.784615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.784596    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sr5w\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.804355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.803657    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"]
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887368    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887453    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887555    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887586    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887629    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sr5w\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887686    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.887775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887714    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.888345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887787    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.888345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.887815    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.889822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.888633    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.889822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.889326    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.889822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.889578    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.890591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.890566    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.891020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.890999    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.893352 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.893335    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.894343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.894319    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.966735 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.966668    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:34.979002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:34.978968    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sr5w\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w\") pod \"e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") " pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:35.000762 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.000682    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:35.455636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.455608    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"]
Nov 03 13:37:35.457512 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:37:35.457476    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c6cece_a343_4fea_9588_851450fa664f.slice/crio-9c950b815ee92be9018ae45bb5fce23d93a9bddfb3807144b380130d847e8a4c WatchSource:0}: Error finding container 9c950b815ee92be9018ae45bb5fce23d93a9bddfb3807144b380130d847e8a4c: Status 404 returned error can't find the container with id 9c950b815ee92be9018ae45bb5fce23d93a9bddfb3807144b380130d847e8a4c
Nov 03 13:37:35.459845 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.459812    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:37:35.459933 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.459880    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:37:35.459933 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.459908    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:37:35.676268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.676230    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" event={"ID":"59c6cece-a343-4fea-9588-851450fa664f","Type":"ContainerStarted","Data":"22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"}
Nov 03 13:37:35.676707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.676284    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" event={"ID":"59c6cece-a343-4fea-9588-851450fa664f","Type":"ContainerStarted","Data":"9c950b815ee92be9018ae45bb5fce23d93a9bddfb3807144b380130d847e8a4c"}
Nov 03 13:37:35.788024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:35.787967    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podStartSLOduration=1.787949501 podStartE2EDuration="1.787949501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:35.785425813 +0000 UTC m=+6397.357009724" watchObservedRunningTime="2026-11-03 13:37:35.787949501 +0000 UTC m=+6397.359533412"
Nov 03 13:37:36.001615 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:36.001574    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:36.006901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:36.006871    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:36.681292 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:36.681260    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:36.682533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:36.682509    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:37:37.651442 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:37:37.651403    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:37:42.316992 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.316938    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"]
Nov 03 13:37:42.320705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.320685    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.362541 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.362506    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.362725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.362704    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rhpn\" (UniqueName: \"kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn\") pod \"payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.369401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.369376    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-dddf906c-dockercfg-j7rpg\""
Nov 03 13:37:42.415611 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.415579    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a6706a\""
Nov 03 13:37:42.419024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.418990    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"]
Nov 03 13:37:42.463225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.463193    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rhpn\" (UniqueName: \"kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn\") pod \"payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.463440 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.463237    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.464179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.464154    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.466511 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.466482    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"]
Nov 03 13:37:42.470551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.470535    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.518305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.518263    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rhpn\" (UniqueName: \"kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn\") pod \"payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.525644 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.525608    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"]
Nov 03 13:37:42.563831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.563786    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.564041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.563889    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8z8f\" (UniqueName: \"kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f\") pod \"payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.633789 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.633670    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:42.664405 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.664367    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.664582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.664419    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8z8f\" (UniqueName: \"kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f\") pod \"payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.665112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.665090    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.719343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.719311    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8z8f\" (UniqueName: \"kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f\") pod \"payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.784359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.784319    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:42.998822 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:42.998717    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"]
Nov 03 13:37:43.001374 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:37:43.001343    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dba1740_c266_4824_b365_1bea2d6da1f5.slice/crio-66407b7c99d9f9d4226c5a3939b36d65de47b559a854310c9f6b8fa9fbb900f5 WatchSource:0}: Error finding container 66407b7c99d9f9d4226c5a3939b36d65de47b559a854310c9f6b8fa9fbb900f5: Status 404 returned error can't find the container with id 66407b7c99d9f9d4226c5a3939b36d65de47b559a854310c9f6b8fa9fbb900f5
Nov 03 13:37:43.158581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.158544    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"]
Nov 03 13:37:43.164806 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:37:43.164779    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0ca15a_db12_4898_a209_3da71eef18d7.slice/crio-40ba72b5263bfa3acb5f2dee00e56549d3acff65edb542d562a7cd29532fc2f7 WatchSource:0}: Error finding container 40ba72b5263bfa3acb5f2dee00e56549d3acff65edb542d562a7cd29532fc2f7: Status 404 returned error can't find the container with id 40ba72b5263bfa3acb5f2dee00e56549d3acff65edb542d562a7cd29532fc2f7
Nov 03 13:37:43.730026 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.729973    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" event={"ID":"2dba1740-c266-4824-b365-1bea2d6da1f5","Type":"ContainerStarted","Data":"1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1"}
Nov 03 13:37:43.730518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.730079    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" event={"ID":"2dba1740-c266-4824-b365-1bea2d6da1f5","Type":"ContainerStarted","Data":"66407b7c99d9f9d4226c5a3939b36d65de47b559a854310c9f6b8fa9fbb900f5"}
Nov 03 13:37:43.730518 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.730111    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:43.731862 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.731838    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" event={"ID":"6d0ca15a-db12-4898-a209-3da71eef18d7","Type":"ContainerStarted","Data":"8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6"}
Nov 03 13:37:43.731946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.731880    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" event={"ID":"6d0ca15a-db12-4898-a209-3da71eef18d7","Type":"ContainerStarted","Data":"40ba72b5263bfa3acb5f2dee00e56549d3acff65edb542d562a7cd29532fc2f7"}
Nov 03 13:37:43.731946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.731915    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:43.839969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.839912    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" podStartSLOduration=1.839894771 podStartE2EDuration="1.839894771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:43.837261001 +0000 UTC m=+6405.408844926" watchObservedRunningTime="2026-11-03 13:37:43.839894771 +0000 UTC m=+6405.411478707"
Nov 03 13:37:43.954833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:43.954730    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" podStartSLOduration=1.954707865 podStartE2EDuration="1.954707865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:43.950581574 +0000 UTC m=+6405.522165489" watchObservedRunningTime="2026-11-03 13:37:43.954707865 +0000 UTC m=+6405.526291778"
Nov 03 13:37:54.740707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:54.740664    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:37:54.741275 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:54.740783    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:37:56.055177 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:56.055138    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"]
Nov 03 13:37:56.102144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:56.102106    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-67rx8"]
Nov 03 13:37:56.571626 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:37:56.571591    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3becd7-955b-43fe-8d60-15bdd31689bf" path="/var/lib/kubelet/pods/bf3becd7-955b-43fe-8d60-15bdd31689bf/volumes"
Nov 03 13:38:00.824128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:00.824097    4546 generic.go:356] "Generic (PLEG): container finished" podID="59c6cece-a343-4fea-9588-851450fa664f" containerID="22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4" exitCode=0
Nov 03 13:38:00.824589 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:00.824175    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" event={"ID":"59c6cece-a343-4fea-9588-851450fa664f","Type":"ContainerDied","Data":"22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"}
Nov 03 13:38:00.824632 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:00.824617    4546 scope.go:122] "RemoveContainer" containerID="22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"
Nov 03 13:38:00.825927 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:00.825900    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:38:00.826030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:00.825971    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:38:00.826030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:00.826004    4546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 13:38:01.839057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:01.839018    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" event={"ID":"59c6cece-a343-4fea-9588-851450fa664f","Type":"ContainerStarted","Data":"5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67"}
Nov 03 13:38:02.001833 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:02.001795    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:38:02.002107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:02.002084    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused" start-of-body=
Nov 03 13:38:02.002215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:02.002134    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused"
Nov 03 13:38:03.001545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:03.001512    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused" start-of-body=
Nov 03 13:38:03.002019 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:03.001565    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused"
Nov 03 13:38:03.303973 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:03.303882    4546 scope.go:122] "RemoveContainer" containerID="f4635f7fb8e00ad6c1c988d07768e2ec64a0a20bb8bc5a0b9c64165c5124d008"
Nov 03 13:38:04.001450 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:04.001409    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused" start-of-body=
Nov 03 13:38:04.001640 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:04.001477    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused"
Nov 03 13:38:05.002150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:05.002112    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused" start-of-body=
Nov 03 13:38:05.002592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:05.002166    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused"
Nov 03 13:38:06.001306 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.001268    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused" start-of-body=
Nov 03 13:38:06.001515 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.001343    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused"
Nov 03 13:38:06.682056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.682019    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:38:06.798104 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.798022    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"]
Nov 03 13:38:06.802510 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.802484    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:06.854137 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.854108    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a6706a\"/\"openshift-service-ca.crt\""
Nov 03 13:38:06.900425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.900384    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a6706a\"/\"kube-root-ca.crt\""
Nov 03 13:38:06.900425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.900384    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a6706a\"/\"simulator-single-main-a6706a-kserve-self-signed-certs\""
Nov 03 13:38:06.900658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.900484    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a6706a\"/\"default-dockercfg-4t4bj\""
Nov 03 13:38:06.902854 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.902833    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"]
Nov 03 13:38:06.941368 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.941331    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:06.941368 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.941372    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:06.941595 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.941560    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:06.941660 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.941641    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:06.941713 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.941694    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdgq\" (UniqueName: \"kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:06.941802 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:06.941785    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.001218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.001174    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused" start-of-body=
Nov 03 13:38:07.001417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.001234    4546 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.151:15021/healthz/ready\": dial tcp 10.133.1.151:15021: connect: connection refused"
Nov 03 13:38:07.043529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.043486    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.043529 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.043538    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.043847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.043618    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.043847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.043675    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.043847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.043723    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgq\" (UniqueName: \"kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.043847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.043799    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.044090 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.044020    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.044417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.044346    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.044417 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.044407    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.046471 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.046446    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.046471 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.046459    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.106201 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.106106    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdgq\" (UniqueName: \"kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq\") pod \"simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") " pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.139889 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.139847    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:07.522586 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.522562    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"]
Nov 03 13:38:07.523501 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:38:07.523476    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59ac1f3a_9144_4767_9265_8e4cd592bf56.slice/crio-2c1d534f0ed93edfa750d7cfce26d9e41c9b9bee1bf916d630011a5eae21f179 WatchSource:0}: Error finding container 2c1d534f0ed93edfa750d7cfce26d9e41c9b9bee1bf916d630011a5eae21f179: Status 404 returned error can't find the container with id 2c1d534f0ed93edfa750d7cfce26d9e41c9b9bee1bf916d630011a5eae21f179
Nov 03 13:38:07.652867 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:38:07.652833    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:38:07.873494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.873399    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8" event={"ID":"59ac1f3a-9144-4767-9265-8e4cd592bf56","Type":"ContainerStarted","Data":"6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b"}
Nov 03 13:38:07.873494 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.873447    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8" event={"ID":"59ac1f3a-9144-4767-9265-8e4cd592bf56","Type":"ContainerStarted","Data":"2c1d534f0ed93edfa750d7cfce26d9e41c9b9bee1bf916d630011a5eae21f179"}
Nov 03 13:38:07.988571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:07.988511    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8" podStartSLOduration=1.988495645 podStartE2EDuration="1.988495645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:07.986054027 +0000 UTC m=+6429.557637957" watchObservedRunningTime="2026-11-03 13:38:07.988495645 +0000 UTC m=+6429.560079590"
Nov 03 13:38:08.006223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:08.006193    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:38:08.007313 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:08.007288    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:38:08.140431 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:08.140336    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:09.045007 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:09.044971    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"]
Nov 03 13:38:09.049397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:09.049369    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vdmhq"]
Nov 03 13:38:10.572105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:10.572073    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f5636a-0641-4ca1-bebb-2c90e9d89bb4" path="/var/lib/kubelet/pods/58f5636a-0641-4ca1-bebb-2c90e9d89bb4/volumes"
Nov 03 13:38:17.140825 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:17.140743    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:17.153878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:17.153847    4546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:17.942225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:17.942191    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:38:37.651343 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:38:37.651300    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:38:43.052156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:43.052126    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"]
Nov 03 13:38:43.099865 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:43.099816    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h8knp"]
Nov 03 13:38:44.572097 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:44.572054    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02691eac-fb44-49a0-b043-709d3721a18a" path="/var/lib/kubelet/pods/02691eac-fb44-49a0-b043-709d3721a18a/volumes"
Nov 03 13:38:53.044825 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.044786    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"]
Nov 03 13:38:53.090611 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.090576    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-psbb5"]
Nov 03 13:38:53.861116 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.861080    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"]
Nov 03 13:38:53.864706 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.864685    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:53.925866 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.925816    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:53.926075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.925933    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:53.926075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.925989    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp65m\" (UniqueName: \"kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:53.962538 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.962495    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:38:53.962538 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.962522    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:38:53.962538 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.962504    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:38:53.962853 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.962558    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:38:53.964741 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:53.964722    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"]
Nov 03 13:38:54.027112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.027081    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp65m\" (UniqueName: \"kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.027346 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.027225    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.027346 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.027265    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.027913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.027893    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.029864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.029842    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.081943 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.081905    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp65m\" (UniqueName: \"kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m\") pod \"maas-api-revoke-sub-d52c211b-jwq5q\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.178559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.178465    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:54.543863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.543836    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"]
Nov 03 13:38:54.545947 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:38:54.545919    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc854e033_3797_4cb6_888b_f9b320db95ef.slice/crio-e50011414968a8e2db4f090fc0459ee0237954ca00ea2a95596dbded630db2d2 WatchSource:0}: Error finding container e50011414968a8e2db4f090fc0459ee0237954ca00ea2a95596dbded630db2d2: Status 404 returned error can't find the container with id e50011414968a8e2db4f090fc0459ee0237954ca00ea2a95596dbded630db2d2
Nov 03 13:38:54.572220 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:54.572195    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14731846-2596-418e-a105-9d1d757641fd" path="/var/lib/kubelet/pods/14731846-2596-418e-a105-9d1d757641fd/volumes"
Nov 03 13:38:55.123401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:55.123302    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q" event={"ID":"c854e033-3797-4cb6-888b-f9b320db95ef","Type":"ContainerStarted","Data":"66da3b5be9e0c2399527b0fbe55d45410b88737718e12707d28e98e4547cca3e"}
Nov 03 13:38:55.123401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:55.123366    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q" event={"ID":"c854e033-3797-4cb6-888b-f9b320db95ef","Type":"ContainerStarted","Data":"e50011414968a8e2db4f090fc0459ee0237954ca00ea2a95596dbded630db2d2"}
Nov 03 13:38:55.276629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:55.276564    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q" podStartSLOduration=2.276548891 podStartE2EDuration="2.276548891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:55.274257045 +0000 UTC m=+6476.845840975" watchObservedRunningTime="2026-11-03 13:38:55.276548891 +0000 UTC m=+6476.848132857"
Nov 03 13:38:56.130835 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:56.130802    4546 generic.go:356] "Generic (PLEG): container finished" podID="c854e033-3797-4cb6-888b-f9b320db95ef" containerID="66da3b5be9e0c2399527b0fbe55d45410b88737718e12707d28e98e4547cca3e" exitCode=0
Nov 03 13:38:56.131330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:56.130892    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q" event={"ID":"c854e033-3797-4cb6-888b-f9b320db95ef","Type":"ContainerDied","Data":"66da3b5be9e0c2399527b0fbe55d45410b88737718e12707d28e98e4547cca3e"}
Nov 03 13:38:57.279701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.279676    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:38:57.365533 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.365495    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m\" (UniqueName: \"kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m\") pod \"c854e033-3797-4cb6-888b-f9b320db95ef\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") "
Nov 03 13:38:57.365720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.365632    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca\") pod \"c854e033-3797-4cb6-888b-f9b320db95ef\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") "
Nov 03 13:38:57.365720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.365698    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca\") pod \"c854e033-3797-4cb6-888b-f9b320db95ef\" (UID: \"c854e033-3797-4cb6-888b-f9b320db95ef\") "
Nov 03 13:38:57.366070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.366042    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca" pod "c854e033-3797-4cb6-888b-f9b320db95ef" (UID: "c854e033-3797-4cb6-888b-f9b320db95ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:57.367876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.367852    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m" pod "c854e033-3797-4cb6-888b-f9b320db95ef" (UID: "c854e033-3797-4cb6-888b-f9b320db95ef"). InnerVolumeSpecName "kube-api-access-pp65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:57.367970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.367894    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca" pod "c854e033-3797-4cb6-888b-f9b320db95ef" (UID: "c854e033-3797-4cb6-888b-f9b320db95ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:38:57.467475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.467443    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:57.467475 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.467475    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c854e033-3797-4cb6-888b-f9b320db95ef-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:57.467690 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:57.467485    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp65m\" (UniqueName: \"kubernetes.io/projected/c854e033-3797-4cb6-888b-f9b320db95ef-kube-api-access-pp65m\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:58.144363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:58.144325    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q" event={"ID":"c854e033-3797-4cb6-888b-f9b320db95ef","Type":"ContainerDied","Data":"e50011414968a8e2db4f090fc0459ee0237954ca00ea2a95596dbded630db2d2"}
Nov 03 13:38:58.144569 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:58.144401    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50011414968a8e2db4f090fc0459ee0237954ca00ea2a95596dbded630db2d2"
Nov 03 13:38:58.144569 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:38:58.144342    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"
Nov 03 13:39:03.351289 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:03.351249    4546 scope.go:122] "RemoveContainer" containerID="041a0a07c10b30730f4e33ae8f3a70660d2ba0ab760cb6e07780f4113390fc6d"
Nov 03 13:39:03.360686 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:03.360661    4546 scope.go:122] "RemoveContainer" containerID="ac1cc6346a9259751e8392d9c1dc34bc471b1aeeab14e891afb69f40b5522f9a"
Nov 03 13:39:07.650696 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:39:07.650656    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:39:10.498261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:10.498222    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"]
Nov 03 13:39:12.670430 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:12.670380    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"]
Nov 03 13:39:12.768087 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:12.768053    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"]
Nov 03 13:39:17.671687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:17.671596    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerName="payload-processing" containerID="cri-o://8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6" gracePeriod=25
Nov 03 13:39:17.769356 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:17.769304    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerName="payload-pre-processing" containerID="cri-o://1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1" gracePeriod=25
Nov 03 13:39:24.738916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:24.738878    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:39:24.739367 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:24.738891    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:39:25.519768 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.519703    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8" podUID="59ac1f3a-9144-4767-9265-8e4cd592bf56" containerName="main" containerID="cri-o://6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b" gracePeriod=45
Nov 03 13:39:25.768797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.768731    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:39:25.952875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.952783    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home\") pod \"59ac1f3a-9144-4767-9265-8e4cd592bf56\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") "
Nov 03 13:39:25.952875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.952853    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache\") pod \"59ac1f3a-9144-4767-9265-8e4cd592bf56\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") "
Nov 03 13:39:25.953105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.952887    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir\") pod \"59ac1f3a-9144-4767-9265-8e4cd592bf56\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") "
Nov 03 13:39:25.953105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.952951    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm\") pod \"59ac1f3a-9144-4767-9265-8e4cd592bf56\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") "
Nov 03 13:39:25.953105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953000    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs\") pod \"59ac1f3a-9144-4767-9265-8e4cd592bf56\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") "
Nov 03 13:39:25.953105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953045    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq\" (UniqueName: \"kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq\") pod \"59ac1f3a-9144-4767-9265-8e4cd592bf56\" (UID: \"59ac1f3a-9144-4767-9265-8e4cd592bf56\") "
Nov 03 13:39:25.953105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953061    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home" pod "59ac1f3a-9144-4767-9265-8e4cd592bf56" (UID: "59ac1f3a-9144-4767-9265-8e4cd592bf56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:25.953379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953190    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache" pod "59ac1f3a-9144-4767-9265-8e4cd592bf56" (UID: "59ac1f3a-9144-4767-9265-8e4cd592bf56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:25.953379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953208    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir" pod "59ac1f3a-9144-4767-9265-8e4cd592bf56" (UID: "59ac1f3a-9144-4767-9265-8e4cd592bf56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:25.953469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953431    4546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-home\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:25.953469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953453    4546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-model-cache\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:25.953564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.953467    4546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-tmp-dir\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:25.955625 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.955589    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs" pod "59ac1f3a-9144-4767-9265-8e4cd592bf56" (UID: "59ac1f3a-9144-4767-9265-8e4cd592bf56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:25.955913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.955887    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm" pod "59ac1f3a-9144-4767-9265-8e4cd592bf56" (UID: "59ac1f3a-9144-4767-9265-8e4cd592bf56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:25.955913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:25.955910    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq" pod "59ac1f3a-9144-4767-9265-8e4cd592bf56" (UID: "59ac1f3a-9144-4767-9265-8e4cd592bf56"). InnerVolumeSpecName "kube-api-access-9sdgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:26.054202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.054160    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdgq\" (UniqueName: \"kubernetes.io/projected/59ac1f3a-9144-4767-9265-8e4cd592bf56-kube-api-access-9sdgq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:26.054202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.054200    4546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac1f3a-9144-4767-9265-8e4cd592bf56-dshm\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:26.054202 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.054210    4546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac1f3a-9144-4767-9265-8e4cd592bf56-tls-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:26.309075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.309037    4546 generic.go:356] "Generic (PLEG): container finished" podID="59ac1f3a-9144-4767-9265-8e4cd592bf56" containerID="6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b" exitCode=0
Nov 03 13:39:26.309317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.309093    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"
Nov 03 13:39:26.309317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.309131    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8" event={"ID":"59ac1f3a-9144-4767-9265-8e4cd592bf56","Type":"ContainerDied","Data":"6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b"}
Nov 03 13:39:26.309317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.309187    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8" event={"ID":"59ac1f3a-9144-4767-9265-8e4cd592bf56","Type":"ContainerDied","Data":"2c1d534f0ed93edfa750d7cfce26d9e41c9b9bee1bf916d630011a5eae21f179"}
Nov 03 13:39:26.309317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.309209    4546 scope.go:122] "RemoveContainer" containerID="6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b"
Nov 03 13:39:26.318954 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.318912    4546 scope.go:122] "RemoveContainer" containerID="6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b"
Nov 03 13:39:26.319211 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:39:26.319189    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b\": container with ID starting with 6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b not found: ID does not exist" containerID="6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b"
Nov 03 13:39:26.319280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.319218    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b"} err="failed to get container status \"6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b\": rpc error: code = NotFound desc = could not find container \"6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b\": container with ID starting with 6e16d4bae6a3285cb17f06ab3cf60d41c25d20b8d113fc0bcfe406ad7ec8119b not found: ID does not exist"
Nov 03 13:39:26.462698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.462655    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"]
Nov 03 13:39:26.466910 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.466878    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a6706a/simulator-single-main-a6706a-kserve-6766ddf69f-wfwv8"]
Nov 03 13:39:26.571905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:26.571826    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ac1f3a-9144-4767-9265-8e4cd592bf56" path="/var/lib/kubelet/pods/59ac1f3a-9144-4767-9265-8e4cd592bf56/volumes"
Nov 03 13:39:28.043722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:28.043688    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"]
Nov 03 13:39:28.091522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:28.091486    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-895kk"]
Nov 03 13:39:28.571687 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:28.571653    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93e3663-a5c7-4875-bddf-49de455f6e6c" path="/var/lib/kubelet/pods/a93e3663-a5c7-4875-bddf-49de455f6e6c/volumes"
Nov 03 13:39:34.738959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:34.738921    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:39:34.739397 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:34.738922    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:39:37.651465 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:39:37.651372    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:39:37.925190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:37.925161    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:39:37.963415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:37.963367    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f\" (UniqueName: \"kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f\") pod \"6d0ca15a-db12-4898-a209-3da71eef18d7\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") "
Nov 03 13:39:37.963630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:37.963573    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume\") pod \"6d0ca15a-db12-4898-a209-3da71eef18d7\" (UID: \"6d0ca15a-db12-4898-a209-3da71eef18d7\") "
Nov 03 13:39:37.964047 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:37.964018    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume" pod "6d0ca15a-db12-4898-a209-3da71eef18d7" (UID: "6d0ca15a-db12-4898-a209-3da71eef18d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:37.966198 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:37.966145    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f" pod "6d0ca15a-db12-4898-a209-3da71eef18d7" (UID: "6d0ca15a-db12-4898-a209-3da71eef18d7"). InnerVolumeSpecName "kube-api-access-x8z8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:38.013613 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.013583    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:39:38.064598 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.064562    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume\") pod \"2dba1740-c266-4824-b365-1bea2d6da1f5\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") "
Nov 03 13:39:38.064868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.064644    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn\" (UniqueName: \"kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn\") pod \"2dba1740-c266-4824-b365-1bea2d6da1f5\" (UID: \"2dba1740-c266-4824-b365-1bea2d6da1f5\") "
Nov 03 13:39:38.064868 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.064862    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0ca15a-db12-4898-a209-3da71eef18d7-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:38.065032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.064881    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8z8f\" (UniqueName: \"kubernetes.io/projected/6d0ca15a-db12-4898-a209-3da71eef18d7-kube-api-access-x8z8f\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:38.065149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.065116    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume" pod "2dba1740-c266-4824-b365-1bea2d6da1f5" (UID: "2dba1740-c266-4824-b365-1bea2d6da1f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:38.067020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.066996    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn" pod "2dba1740-c266-4824-b365-1bea2d6da1f5" (UID: "2dba1740-c266-4824-b365-1bea2d6da1f5"). InnerVolumeSpecName "kube-api-access-6rhpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:38.165497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.165400    4546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dba1740-c266-4824-b365-1bea2d6da1f5-plugins-config-volume\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:38.165497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.165432    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rhpn\" (UniqueName: \"kubernetes.io/projected/2dba1740-c266-4824-b365-1bea2d6da1f5-kube-api-access-6rhpn\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:38.374984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.374946    4546 generic.go:356] "Generic (PLEG): container finished" podID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerID="8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6" exitCode=0
Nov 03 13:39:38.375200 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.375000    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"
Nov 03 13:39:38.375200 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.375032    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" event={"ID":"6d0ca15a-db12-4898-a209-3da71eef18d7","Type":"ContainerDied","Data":"8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6"}
Nov 03 13:39:38.375200 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.375090    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n" event={"ID":"6d0ca15a-db12-4898-a209-3da71eef18d7","Type":"ContainerDied","Data":"40ba72b5263bfa3acb5f2dee00e56549d3acff65edb542d562a7cd29532fc2f7"}
Nov 03 13:39:38.375200 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.375120    4546 scope.go:122] "RemoveContainer" containerID="8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6"
Nov 03 13:39:38.376576 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.376554    4546 generic.go:356] "Generic (PLEG): container finished" podID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerID="1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1" exitCode=0
Nov 03 13:39:38.376718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.376600    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"
Nov 03 13:39:38.376718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.376639    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" event={"ID":"2dba1740-c266-4824-b365-1bea2d6da1f5","Type":"ContainerDied","Data":"1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1"}
Nov 03 13:39:38.376718 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.376689    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t" event={"ID":"2dba1740-c266-4824-b365-1bea2d6da1f5","Type":"ContainerDied","Data":"66407b7c99d9f9d4226c5a3939b36d65de47b559a854310c9f6b8fa9fbb900f5"}
Nov 03 13:39:38.385669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.385645    4546 scope.go:122] "RemoveContainer" containerID="8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6"
Nov 03 13:39:38.386037 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:39:38.386013    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6\": container with ID starting with 8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6 not found: ID does not exist" containerID="8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6"
Nov 03 13:39:38.386163 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.386043    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6"} err="failed to get container status \"8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6\": rpc error: code = NotFound desc = could not find container \"8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6\": container with ID starting with 8fba6e0f87c6226ac39b2b75f6e0279fddb9b22cc9a22d12841e0611b9853fa6 not found: ID does not exist"
Nov 03 13:39:38.386163 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.386061    4546 scope.go:122] "RemoveContainer" containerID="1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1"
Nov 03 13:39:38.394630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.394609    4546 scope.go:122] "RemoveContainer" containerID="1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1"
Nov 03 13:39:38.395070 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:39:38.395048    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1\": container with ID starting with 1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1 not found: ID does not exist" containerID="1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1"
Nov 03 13:39:38.395149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.395076    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1"} err="failed to get container status \"1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1\": rpc error: code = NotFound desc = could not find container \"1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1\": container with ID starting with 1e2236b73b61da0ef2450c2b72661c53ff396ddacd52b87133cbb708ed7dd9c1 not found: ID does not exist"
Nov 03 13:39:38.527465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.527429    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"]
Nov 03 13:39:38.573590 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.573541    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a6706a-5db8cdfb8qft2n"]
Nov 03 13:39:38.632928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.632892    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"]
Nov 03 13:39:38.637430 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:38.637394    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a6706a-69fc72qv5t"]
Nov 03 13:39:40.571564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:40.571531    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" path="/var/lib/kubelet/pods/2dba1740-c266-4824-b365-1bea2d6da1f5/volumes"
Nov 03 13:39:40.572063 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:40.572048    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" path="/var/lib/kubelet/pods/6d0ca15a-db12-4898-a209-3da71eef18d7/volumes"
Nov 03 13:39:50.046243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:50.046203    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"]
Nov 03 13:39:50.094221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:50.094182    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hb9h5"]
Nov 03 13:39:50.572228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:39:50.572195    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d3b2e3-29be-4fa9-a459-9a8942aada6c" path="/var/lib/kubelet/pods/96d3b2e3-29be-4fa9-a459-9a8942aada6c/volumes"
Nov 03 13:40:00.953126 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953085    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"]
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953475    4546 cpu_manager.go:414] "containerMap: removing container" podUID="59ac1f3a-9144-4767-9265-8e4cd592bf56" containerName="main"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953486    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ac1f3a-9144-4767-9265-8e4cd592bf56" containerName="main"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953498    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c854e033-3797-4cb6-888b-f9b320db95ef" containerName="revoke-keys"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953503    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c854e033-3797-4cb6-888b-f9b320db95ef" containerName="revoke-keys"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953520    4546 cpu_manager.go:414] "containerMap: removing container" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerName="payload-processing"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953527    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerName="payload-processing"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953541    4546 cpu_manager.go:414] "containerMap: removing container" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerName="payload-pre-processing"
Nov 03 13:40:00.953551 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953546    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerName="payload-pre-processing"
Nov 03 13:40:00.953870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953603    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0ca15a-db12-4898-a209-3da71eef18d7" containerName="payload-processing"
Nov 03 13:40:00.953870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953612    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dba1740-c266-4824-b365-1bea2d6da1f5" containerName="payload-pre-processing"
Nov 03 13:40:00.953870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953622    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ac1f3a-9144-4767-9265-8e4cd592bf56" containerName="main"
Nov 03 13:40:00.953870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.953629    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c854e033-3797-4cb6-888b-f9b320db95ef" containerName="revoke-keys"
Nov 03 13:40:00.957172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:00.957148    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.006605 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.006571    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:40:01.006836 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.006619    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:40:01.053167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.053122    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:40:01.053167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.053133    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:40:01.055358 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.055329    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"]
Nov 03 13:40:01.094990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.094953    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kbx\" (UniqueName: \"kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.094990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.094994    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.095215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.095021    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.196051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.196013    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kbx\" (UniqueName: \"kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.196051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.196052    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.196337 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.196169    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.196777 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.196719    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.198760 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.198729    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.250218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.250183    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kbx\" (UniqueName: \"kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.269834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.269801    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:01.635527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:01.635487    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"]
Nov 03 13:40:01.637070 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:40:01.637037    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc9b2d9_cfeb_4b4f_a082_5b3a46eba490.slice/crio-aec062b2d04553105d61a1ac6c77ffc53eb3ab7040fb881cf58cf58c39765309 WatchSource:0}: Error finding container aec062b2d04553105d61a1ac6c77ffc53eb3ab7040fb881cf58cf58c39765309: Status 404 returned error can't find the container with id aec062b2d04553105d61a1ac6c77ffc53eb3ab7040fb881cf58cf58c39765309
Nov 03 13:40:02.504540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:02.504495    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m" event={"ID":"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490","Type":"ContainerStarted","Data":"077ea542f8081ed928e8ac85c8f015145ac16c515026f1b4c61314bb053091d4"}
Nov 03 13:40:02.504986 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:02.504561    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m" event={"ID":"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490","Type":"ContainerStarted","Data":"aec062b2d04553105d61a1ac6c77ffc53eb3ab7040fb881cf58cf58c39765309"}
Nov 03 13:40:02.614391 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:02.614324    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m" podStartSLOduration=2.614311949 podStartE2EDuration="2.614311949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:02.613018264 +0000 UTC m=+6544.184602189" watchObservedRunningTime="2026-11-03 13:40:02.614311949 +0000 UTC m=+6544.185895903"
Nov 03 13:40:03.387258 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:03.387228    4546 scope.go:122] "RemoveContainer" containerID="2dffaeaf45bc471d12bf0b1ae687bf2d8cb8a5bdde5fdc1f7e8cc07bf0246e45"
Nov 03 13:40:03.396024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:03.396001    4546 scope.go:122] "RemoveContainer" containerID="260af890e71273be8326e848479f8230dd15b5ed947e41e20e8772d9bccc01a1"
Nov 03 13:40:03.404263 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:03.404236    4546 scope.go:122] "RemoveContainer" containerID="a4752ad06f3cb76b9e246098cbc479c3bcbace776b358631ef6223c55c5d475a"
Nov 03 13:40:03.511416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:03.511378    4546 generic.go:356] "Generic (PLEG): container finished" podID="fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" containerID="077ea542f8081ed928e8ac85c8f015145ac16c515026f1b4c61314bb053091d4" exitCode=0
Nov 03 13:40:03.511937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:03.511457    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m" event={"ID":"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490","Type":"ContainerDied","Data":"077ea542f8081ed928e8ac85c8f015145ac16c515026f1b4c61314bb053091d4"}
Nov 03 13:40:04.661383 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.661354    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:04.833443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.833354    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx\" (UniqueName: \"kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx\") pod \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") "
Nov 03 13:40:04.833443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.833398    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca\") pod \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") "
Nov 03 13:40:04.833655 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.833445    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca\") pod \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\" (UID: \"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490\") "
Nov 03 13:40:04.833929 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.833905    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca" pod "fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" (UID: "fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:04.835770 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.835735    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx" pod "fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" (UID: "fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490"). InnerVolumeSpecName "kube-api-access-l6kbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:04.835884 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.835820    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca" pod "fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" (UID: "fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:04.935120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.935078    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6kbx\" (UniqueName: \"kubernetes.io/projected/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-kube-api-access-l6kbx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:04.935120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.935114    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:04.935120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:04.935126    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:05.523420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:05.523383    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"
Nov 03 13:40:05.523420 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:05.523401    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m" event={"ID":"fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490","Type":"ContainerDied","Data":"aec062b2d04553105d61a1ac6c77ffc53eb3ab7040fb881cf58cf58c39765309"}
Nov 03 13:40:05.523655 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:05.523459    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec062b2d04553105d61a1ac6c77ffc53eb3ab7040fb881cf58cf58c39765309"
Nov 03 13:40:07.650813 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:40:07.650776    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:40:11.050376 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:11.050336    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"]
Nov 03 13:40:11.096167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:11.096133    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r48jj"]
Nov 03 13:40:12.571594 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:12.571549    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30497586-d966-4352-820e-aae658820eee" path="/var/lib/kubelet/pods/30497586-d966-4352-820e-aae658820eee/volumes"
Nov 03 13:40:20.966464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:20.966426    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"]
Nov 03 13:40:21.012461 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:21.012423    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a6706a-ssq7m"]
Nov 03 13:40:21.169462 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:21.169431    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"]
Nov 03 13:40:21.169798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:21.169689    4546 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" containerID="cri-o://5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67" gracePeriod=30
Nov 03 13:40:22.571834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.571799    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" path="/var/lib/kubelet/pods/fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490/volumes"
Nov 03 13:40:22.837469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.837368    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"]
Nov 03 13:40:22.883480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.883435    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rvgdg"]
Nov 03 13:40:22.954704 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.954671    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"]
Nov 03 13:40:22.955156 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.955137    4546 cpu_manager.go:414] "containerMap: removing container" podUID="fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" containerName="revoke-keys"
Nov 03 13:40:22.955259 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.955158    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" containerName="revoke-keys"
Nov 03 13:40:22.955326 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.955306    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc9b2d9-cfeb-4b4f-a082-5b3a46eba490" containerName="revoke-keys"
Nov 03 13:40:22.958536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:22.958513    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.007650 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.007594    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:40:23.007650 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.007605    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:40:23.007650 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.007633    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:40:23.007982 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.007699    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:40:23.009141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.009116    4546 patch_prober.go:29] interesting pod/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:40:23.009265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.009158    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:40:23.012341 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.012313    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhv2v\" (UniqueName: \"kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.012476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.012443    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.012548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.012501    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.013358 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.013333    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"]
Nov 03 13:40:23.113696 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.113591    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.113696 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.113649    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.114014 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.113794    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhv2v\" (UniqueName: \"kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.114303 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.114280    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.116484 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.116464    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.171406 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.171372    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhv2v\" (UniqueName: \"kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v\") pod \"maas-api-revoke-sub-3d4156f1-vm2pj\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.271210 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.271165    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:23.644856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:23.644831    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"]
Nov 03 13:40:23.647380 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:40:23.647345    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebb8fa6_3c8f_43a9_b365_560bcb357d30.slice/crio-a54a161057289a1ce54772869066ac7943f5b088a316f4a0b17af5e3a782221e WatchSource:0}: Error finding container a54a161057289a1ce54772869066ac7943f5b088a316f4a0b17af5e3a782221e: Status 404 returned error can't find the container with id a54a161057289a1ce54772869066ac7943f5b088a316f4a0b17af5e3a782221e
Nov 03 13:40:24.572443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:24.572408    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b801e1-162e-4b42-b092-ef567a5ba407" path="/var/lib/kubelet/pods/34b801e1-162e-4b42-b092-ef567a5ba407/volumes"
Nov 03 13:40:24.622886 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:24.622849    4546 generic.go:356] "Generic (PLEG): container finished" podID="3ebb8fa6-3c8f-43a9-b365-560bcb357d30" containerID="02e4a19246682244a406fffe4b4fe0dbd74dcf03a92f65ab204d26842fc2d73a" exitCode=0
Nov 03 13:40:24.623101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:24.622918    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj" event={"ID":"3ebb8fa6-3c8f-43a9-b365-560bcb357d30","Type":"ContainerDied","Data":"02e4a19246682244a406fffe4b4fe0dbd74dcf03a92f65ab204d26842fc2d73a"}
Nov 03 13:40:24.623101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:24.622969    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj" event={"ID":"3ebb8fa6-3c8f-43a9-b365-560bcb357d30","Type":"ContainerStarted","Data":"a54a161057289a1ce54772869066ac7943f5b088a316f4a0b17af5e3a782221e"}
Nov 03 13:40:25.768054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.768027    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:25.841940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.841907    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca\") pod \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") "
Nov 03 13:40:25.842153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.841977    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca\") pod \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") "
Nov 03 13:40:25.842153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.842082    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v\" (UniqueName: \"kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v\") pod \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\" (UID: \"3ebb8fa6-3c8f-43a9-b365-560bcb357d30\") "
Nov 03 13:40:25.842290 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.842272    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca" pod "3ebb8fa6-3c8f-43a9-b365-560bcb357d30" (UID: "3ebb8fa6-3c8f-43a9-b365-560bcb357d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:25.842422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.842405    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:25.844237 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.844211    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v" pod "3ebb8fa6-3c8f-43a9-b365-560bcb357d30" (UID: "3ebb8fa6-3c8f-43a9-b365-560bcb357d30"). InnerVolumeSpecName "kube-api-access-hhv2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:25.844330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.844315    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca" pod "3ebb8fa6-3c8f-43a9-b365-560bcb357d30" (UID: "3ebb8fa6-3c8f-43a9-b365-560bcb357d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:25.943719 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.943625    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:25.943719 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:25.943658    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhv2v\" (UniqueName: \"kubernetes.io/projected/3ebb8fa6-3c8f-43a9-b365-560bcb357d30-kube-api-access-hhv2v\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.411446 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.411419    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:40:26.447858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447741    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.447858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447847    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447877    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447902    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447930    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447953    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.447990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448132 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448080    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:26.448472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448302    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448354    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:26.448472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448374    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data\") pod \"59c6cece-a343-4fea-9588-851450fa664f\" (UID: \"59c6cece-a343-4fea-9588-851450fa664f\") "
Nov 03 13:40:26.448472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448386    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:26.448709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448660    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:26.448880 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.448781    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:26.449150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.449084    4546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-credential-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.449150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.449109    4546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-socket\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.449150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.449125    4546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c6cece-a343-4fea-9588-851450fa664f-istiod-ca-cert\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.449150 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.449143    4546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-data\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.449394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.449157    4546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-workload-certs\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.450474 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.450444    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:26.450596 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.450526    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:40:26.450596 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.450550    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "kube-api-access-8sr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:26.450721 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.450663    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token" pod "59c6cece-a343-4fea-9588-851450fa664f" (UID: "59c6cece-a343-4fea-9588-851450fa664f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:26.549710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.549675    4546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-istio-token\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.549710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.549703    4546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c6cece-a343-4fea-9588-851450fa664f-istio-podinfo\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.549710 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.549714    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sr5w\" (UniqueName: \"kubernetes.io/projected/59c6cece-a343-4fea-9588-851450fa664f-kube-api-access-8sr5w\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.549969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.549724    4546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c6cece-a343-4fea-9588-851450fa664f-istio-envoy\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:26.633597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.633562    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"
Nov 03 13:40:26.633597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.633573    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj" event={"ID":"3ebb8fa6-3c8f-43a9-b365-560bcb357d30","Type":"ContainerDied","Data":"a54a161057289a1ce54772869066ac7943f5b088a316f4a0b17af5e3a782221e"}
Nov 03 13:40:26.633912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.633615    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a54a161057289a1ce54772869066ac7943f5b088a316f4a0b17af5e3a782221e"
Nov 03 13:40:26.635454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.635431    4546 generic.go:356] "Generic (PLEG): container finished" podID="59c6cece-a343-4fea-9588-851450fa664f" containerID="5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67" exitCode=0
Nov 03 13:40:26.635599 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.635478    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"
Nov 03 13:40:26.635599 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.635583    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" event={"ID":"59c6cece-a343-4fea-9588-851450fa664f","Type":"ContainerDied","Data":"5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67"}
Nov 03 13:40:26.635724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.635630    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w" event={"ID":"59c6cece-a343-4fea-9588-851450fa664f","Type":"ContainerDied","Data":"9c950b815ee92be9018ae45bb5fce23d93a9bddfb3807144b380130d847e8a4c"}
Nov 03 13:40:26.635724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.635663    4546 scope.go:122] "RemoveContainer" containerID="5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67"
Nov 03 13:40:26.645743 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.645718    4546 scope.go:122] "RemoveContainer" containerID="22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"
Nov 03 13:40:26.654643 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.654620    4546 scope.go:122] "RemoveContainer" containerID="5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67"
Nov 03 13:40:26.654982 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:40:26.654964    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67\": container with ID starting with 5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67 not found: ID does not exist" containerID="5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67"
Nov 03 13:40:26.655045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.654994    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67"} err="failed to get container status \"5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67\": rpc error: code = NotFound desc = could not find container \"5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67\": container with ID starting with 5644913c48157a0705ebafc021ef02efdd815627d7aad1833a6722794e401c67 not found: ID does not exist"
Nov 03 13:40:26.655045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.655013    4546 scope.go:122] "RemoveContainer" containerID="22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"
Nov 03 13:40:26.655316 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:40:26.655297    4546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4\": container with ID starting with 22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4 not found: ID does not exist" containerID="22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"
Nov 03 13:40:26.655372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.655323    4546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4"} err="failed to get container status \"22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4\": rpc error: code = NotFound desc = could not find container \"22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4\": container with ID starting with 22cba2ce409545ec23c77696d542ef6a146fd5c4c377baac65f8ab42fa9eafa4 not found: ID does not exist"
Nov 03 13:40:26.789946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.789906    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"]
Nov 03 13:40:26.794068 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:26.794037    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a6706a-openshift-default-85d7b87cbfb9t5w"]
Nov 03 13:40:28.572733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:28.572686    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c6cece-a343-4fea-9588-851450fa664f" path="/var/lib/kubelet/pods/59c6cece-a343-4fea-9588-851450fa664f/volumes"
Nov 03 13:40:29.045138 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:29.045103    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"]
Nov 03 13:40:29.091066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:29.091027    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-22qzs"]
Nov 03 13:40:30.572044 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:40:30.572009    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18400f5e-213f-4ee3-9981-d17ab5820b7d" path="/var/lib/kubelet/pods/18400f5e-213f-4ee3-9981-d17ab5820b7d/volumes"
Nov 03 13:40:37.652243 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:40:37.652191    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:41:03.444197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:41:03.444162    4546 scope.go:122] "RemoveContainer" containerID="d2bd462c7a8520e142abc98d204a7ead6328a0fc6444fc2328adf31baa453f7a"
Nov 03 13:41:03.453098 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:41:03.453076    4546 scope.go:122] "RemoveContainer" containerID="35f754029e6d268285bf5f9047a3585a9b3925749dff6153ba56fc083ac3ae93"
Nov 03 13:41:07.650627 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:41:07.650533    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:41:37.650582 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:41:37.650541    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:41:49.047679 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:41:49.047643    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"]
Nov 03 13:41:49.093560 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:41:49.093525    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2cx8d"]
Nov 03 13:41:50.571364 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:41:50.571327    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d45e8b-e78d-4f5b-9c04-b476921947b0" path="/var/lib/kubelet/pods/d6d45e8b-e78d-4f5b-9c04-b476921947b0/volumes"
Nov 03 13:42:00.046847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:00.046807    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"]
Nov 03 13:42:00.093035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:00.093001    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sd7gk"]
Nov 03 13:42:00.572283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:00.572248    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08684a72-b72d-42b0-bcb2-5131fed86624" path="/var/lib/kubelet/pods/08684a72-b72d-42b0-bcb2-5131fed86624/volumes"
Nov 03 13:42:03.481102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:03.481069    4546 scope.go:122] "RemoveContainer" containerID="eeea41580113a80abd8a4648feeab8992388eeb398c740003fc75d3d57226823"
Nov 03 13:42:03.490499 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:03.490476    4546 scope.go:122] "RemoveContainer" containerID="4c05847a2eced35ba04da63ea751df8c0c8a5c7fe717d6d537e6f29874c3f287"
Nov 03 13:42:03.499318 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:03.499290    4546 scope.go:122] "RemoveContainer" containerID="4fbaa32d6ef6b7529904148e9d6e30731eae3bdd1c321c005a1fef2c6ace2bd7"
Nov 03 13:42:03.507956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:03.507930    4546 scope.go:122] "RemoveContainer" containerID="080f567e58b9ffe7690cea2e712d87224ebb8a0496e504b45c912b3ce41adc44"
Nov 03 13:42:07.700680 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:42:07.700642    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:42:10.322345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322308    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"]
Nov 03 13:42:10.322932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322679    4546 cpu_manager.go:414] "containerMap: removing container" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy"
Nov 03 13:42:10.322932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322689    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy"
Nov 03 13:42:10.322932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322726    4546 cpu_manager.go:414] "containerMap: removing container" podUID="3ebb8fa6-3c8f-43a9-b365-560bcb357d30" containerName="revoke-keys"
Nov 03 13:42:10.322932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322731    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ebb8fa6-3c8f-43a9-b365-560bcb357d30" containerName="revoke-keys"
Nov 03 13:42:10.322932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322803    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy"
Nov 03 13:42:10.322932 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.322813    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ebb8fa6-3c8f-43a9-b365-560bcb357d30" containerName="revoke-keys"
Nov 03 13:42:10.327492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.327460    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.376922 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.376888    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:42:10.405526 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.405486    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.405738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.405559    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.405738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.405685    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9s4v\" (UniqueName: \"kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.422928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.422889    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:42:10.422928 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.422889    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:42:10.423149 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.422949    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:42:10.424876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.424856    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"]
Nov 03 13:42:10.506555 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.506517    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9s4v\" (UniqueName: \"kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.506831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.506628    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.506831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.506703    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.507377 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.507347    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.509399 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.509372    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.561155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.561119    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9s4v\" (UniqueName: \"kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v\") pod \"maas-api-revoke-sub-d576e68b-glvcc\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:10.640967 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:10.640871    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:11.005862 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:11.005832    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"]
Nov 03 13:42:11.008162 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:42:11.008137    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5d3b74_c7c3_450e_846a_3a5989926d59.slice/crio-536d6515ba07064095935db21077cf4861ab8e10234b965d1c45273a6862d422 WatchSource:0}: Error finding container 536d6515ba07064095935db21077cf4861ab8e10234b965d1c45273a6862d422: Status 404 returned error can't find the container with id 536d6515ba07064095935db21077cf4861ab8e10234b965d1c45273a6862d422
Nov 03 13:42:11.190301 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:11.190264    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc" event={"ID":"bd5d3b74-c7c3-450e-846a-3a5989926d59","Type":"ContainerStarted","Data":"536d6515ba07064095935db21077cf4861ab8e10234b965d1c45273a6862d422"}
Nov 03 13:42:12.204056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:12.203960    4546 generic.go:356] "Generic (PLEG): container finished" podID="bd5d3b74-c7c3-450e-846a-3a5989926d59" containerID="46866e009cfa81a8e8ae502a625f06a877a517a0128370064d45199fc8572be8" exitCode=0
Nov 03 13:42:12.204056 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:12.204041    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc" event={"ID":"bd5d3b74-c7c3-450e-846a-3a5989926d59","Type":"ContainerDied","Data":"46866e009cfa81a8e8ae502a625f06a877a517a0128370064d45199fc8572be8"}
Nov 03 13:42:13.350864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.350826    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:13.434724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.434680    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca\") pod \"bd5d3b74-c7c3-450e-846a-3a5989926d59\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") "
Nov 03 13:42:13.434977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.434793    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v\" (UniqueName: \"kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v\") pod \"bd5d3b74-c7c3-450e-846a-3a5989926d59\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") "
Nov 03 13:42:13.434977 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.434822    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca\") pod \"bd5d3b74-c7c3-450e-846a-3a5989926d59\" (UID: \"bd5d3b74-c7c3-450e-846a-3a5989926d59\") "
Nov 03 13:42:13.435235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.435206    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca" pod "bd5d3b74-c7c3-450e-846a-3a5989926d59" (UID: "bd5d3b74-c7c3-450e-846a-3a5989926d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:13.437129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.437098    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca" pod "bd5d3b74-c7c3-450e-846a-3a5989926d59" (UID: "bd5d3b74-c7c3-450e-846a-3a5989926d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:13.437129 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.437117    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v" pod "bd5d3b74-c7c3-450e-846a-3a5989926d59" (UID: "bd5d3b74-c7c3-450e-846a-3a5989926d59"). InnerVolumeSpecName "kube-api-access-q9s4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:13.536084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.536042    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:13.536084 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.536093    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9s4v\" (UniqueName: \"kubernetes.io/projected/bd5d3b74-c7c3-450e-846a-3a5989926d59-kube-api-access-q9s4v\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:13.536340 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:13.536108    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd5d3b74-c7c3-450e-846a-3a5989926d59-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:14.215507 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:14.215477    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"
Nov 03 13:42:14.215705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:14.215477    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc" event={"ID":"bd5d3b74-c7c3-450e-846a-3a5989926d59","Type":"ContainerDied","Data":"536d6515ba07064095935db21077cf4861ab8e10234b965d1c45273a6862d422"}
Nov 03 13:42:14.215705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:14.215606    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536d6515ba07064095935db21077cf4861ab8e10234b965d1c45273a6862d422"
Nov 03 13:42:30.473987 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:30.473932    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:42:30.475550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:30.475509    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:42:32.059563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.059529    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"]
Nov 03 13:42:32.060002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.059935    4546 cpu_manager.go:414] "containerMap: removing container" podUID="bd5d3b74-c7c3-450e-846a-3a5989926d59" containerName="revoke-keys"
Nov 03 13:42:32.060002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.059948    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5d3b74-c7c3-450e-846a-3a5989926d59" containerName="revoke-keys"
Nov 03 13:42:32.060002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.059971    4546 cpu_manager.go:414] "containerMap: removing container" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy"
Nov 03 13:42:32.060002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.059977    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy"
Nov 03 13:42:32.060147 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.060067    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c6cece-a343-4fea-9588-851450fa664f" containerName="istio-proxy"
Nov 03 13:42:32.060147 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.060075    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5d3b74-c7c3-450e-846a-3a5989926d59" containerName="revoke-keys"
Nov 03 13:42:32.063357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.063333    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.092779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.092732    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.092970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.092880    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxblp\" (UniqueName: \"kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.092970 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.092911    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.112266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.112223    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:42:32.158465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.158433    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:42:32.158465 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.158447    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:42:32.158697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.158546    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:42:32.160492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.160467    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"]
Nov 03 13:42:32.193600 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.193559    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxblp\" (UniqueName: \"kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.193881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.193618    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.193881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.193714    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.194550 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.194523    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.196359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.196335    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.247593 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.247558    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxblp\" (UniqueName: \"kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp\") pod \"maas-api-revoke-sub-d2bc93bc-nk7gh\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.376692 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.376596    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:32.740058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:32.740035    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"]
Nov 03 13:42:32.742231 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:42:32.742201    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd700e5dd_004d_4561_b77c_e12fc14854d6.slice/crio-aaef64a2be603bcccbfce23a093900c51c447a93b02fcb6c05dea75db2bd38e2 WatchSource:0}: Error finding container aaef64a2be603bcccbfce23a093900c51c447a93b02fcb6c05dea75db2bd38e2: Status 404 returned error can't find the container with id aaef64a2be603bcccbfce23a093900c51c447a93b02fcb6c05dea75db2bd38e2
Nov 03 13:42:33.319540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:33.319440    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh" event={"ID":"d700e5dd-004d-4561-b77c-e12fc14854d6","Type":"ContainerStarted","Data":"1814a8cbf27cd1dc2a6da54b9ed99548d6e7672797d7cdf31705e3a6f457d376"}
Nov 03 13:42:33.319540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:33.319528    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh" event={"ID":"d700e5dd-004d-4561-b77c-e12fc14854d6","Type":"ContainerStarted","Data":"aaef64a2be603bcccbfce23a093900c51c447a93b02fcb6c05dea75db2bd38e2"}
Nov 03 13:42:33.427351 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:33.427288    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh" podStartSLOduration=1.427272442 podStartE2EDuration="1.427272442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:33.425336844 +0000 UTC m=+6694.996920757" watchObservedRunningTime="2026-11-03 13:42:33.427272442 +0000 UTC m=+6694.998856354"
Nov 03 13:42:34.325469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:34.325435    4546 generic.go:356] "Generic (PLEG): container finished" podID="d700e5dd-004d-4561-b77c-e12fc14854d6" containerID="1814a8cbf27cd1dc2a6da54b9ed99548d6e7672797d7cdf31705e3a6f457d376" exitCode=0
Nov 03 13:42:34.325915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:34.325484    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh" event={"ID":"d700e5dd-004d-4561-b77c-e12fc14854d6","Type":"ContainerDied","Data":"1814a8cbf27cd1dc2a6da54b9ed99548d6e7672797d7cdf31705e3a6f457d376"}
Nov 03 13:42:35.469629 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.469598    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:35.619480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.619383    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca\") pod \"d700e5dd-004d-4561-b77c-e12fc14854d6\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") "
Nov 03 13:42:35.619480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.619451    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp\" (UniqueName: \"kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp\") pod \"d700e5dd-004d-4561-b77c-e12fc14854d6\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") "
Nov 03 13:42:35.619689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.619489    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca\") pod \"d700e5dd-004d-4561-b77c-e12fc14854d6\" (UID: \"d700e5dd-004d-4561-b77c-e12fc14854d6\") "
Nov 03 13:42:35.619959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.619926    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca" pod "d700e5dd-004d-4561-b77c-e12fc14854d6" (UID: "d700e5dd-004d-4561-b77c-e12fc14854d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:35.621946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.621919    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp" pod "d700e5dd-004d-4561-b77c-e12fc14854d6" (UID: "d700e5dd-004d-4561-b77c-e12fc14854d6"). InnerVolumeSpecName "kube-api-access-rxblp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:35.622043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.621973    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca" pod "d700e5dd-004d-4561-b77c-e12fc14854d6" (UID: "d700e5dd-004d-4561-b77c-e12fc14854d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:35.721228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.721191    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:35.721228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.721221    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxblp\" (UniqueName: \"kubernetes.io/projected/d700e5dd-004d-4561-b77c-e12fc14854d6-kube-api-access-rxblp\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:35.721228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:35.721234    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e5dd-004d-4561-b77c-e12fc14854d6-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:36.337444 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:36.337408    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh" event={"ID":"d700e5dd-004d-4561-b77c-e12fc14854d6","Type":"ContainerDied","Data":"aaef64a2be603bcccbfce23a093900c51c447a93b02fcb6c05dea75db2bd38e2"}
Nov 03 13:42:36.337444 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:36.337446    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"
Nov 03 13:42:36.337738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:42:36.337457    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaef64a2be603bcccbfce23a093900c51c447a93b02fcb6c05dea75db2bd38e2"
Nov 03 13:42:37.651818 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:42:37.651723    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:43:00.480776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:00.480327    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hsknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:43:00.480776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:00.480431    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hsknc" podUID="d962e47d-d4cc-4fa6-b010-1ebf687b3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:43:03.540656 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:03.540617    4546 scope.go:122] "RemoveContainer" containerID="b86bc6a40bd0c998c227722f8da47855ab560621906ad8210095740d63fdf362"
Nov 03 13:43:03.549941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:03.549916    4546 scope.go:122] "RemoveContainer" containerID="a54f35c114f6ae2237eebace58b9b0898f61d00e09e5bc1343eabf0bd8a76546"
Nov 03 13:43:07.650183 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:43:07.650143    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:43:10.911805 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.911762    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"]
Nov 03 13:43:10.912232 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.912167    4546 cpu_manager.go:414] "containerMap: removing container" podUID="d700e5dd-004d-4561-b77c-e12fc14854d6" containerName="revoke-keys"
Nov 03 13:43:10.912232 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.912178    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d700e5dd-004d-4561-b77c-e12fc14854d6" containerName="revoke-keys"
Nov 03 13:43:10.912312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.912247    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="d700e5dd-004d-4561-b77c-e12fc14854d6" containerName="revoke-keys"
Nov 03 13:43:10.916782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.916737    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:10.926708 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.926678    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:10.926888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.926724    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnk2p\" (UniqueName: \"kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:10.926955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.926882    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:10.967491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.967451    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:43:10.967491 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:10.967462    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:43:11.013797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.013730    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:43:11.014009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.013888    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:43:11.015896 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.015871    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"]
Nov 03 13:43:11.027729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.027683    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.027957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.027769    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnk2p\" (UniqueName: \"kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.027957 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.027833    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.028480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.028459    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.030564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.030539    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.081902 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.081867    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnk2p\" (UniqueName: \"kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p\") pod \"maas-api-revoke-sub-97ceb754-krnhp\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.229545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.229502    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:11.595451 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.595421    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"]
Nov 03 13:43:11.598160 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:43:11.598085    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4756ee1e_98ec_4841_a9e1_dd6b04383dab.slice/crio-884edc9e9157660d151935f6bd8d207562fa6fe16ce6e67f2fd1bc25c7c7a06a WatchSource:0}: Error finding container 884edc9e9157660d151935f6bd8d207562fa6fe16ce6e67f2fd1bc25c7c7a06a: Status 404 returned error can't find the container with id 884edc9e9157660d151935f6bd8d207562fa6fe16ce6e67f2fd1bc25c7c7a06a
Nov 03 13:43:11.612548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:11.612518    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp" event={"ID":"4756ee1e-98ec-4841-a9e1-dd6b04383dab","Type":"ContainerStarted","Data":"884edc9e9157660d151935f6bd8d207562fa6fe16ce6e67f2fd1bc25c7c7a06a"}
Nov 03 13:43:12.620011 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:12.619971    4546 generic.go:356] "Generic (PLEG): container finished" podID="4756ee1e-98ec-4841-a9e1-dd6b04383dab" containerID="d035d99cdb444a084d24459b9eb19bf8c15f4f43c0add9f5d5f2f4a501be387e" exitCode=0
Nov 03 13:43:12.620525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:12.620033    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp" event={"ID":"4756ee1e-98ec-4841-a9e1-dd6b04383dab","Type":"ContainerDied","Data":"d035d99cdb444a084d24459b9eb19bf8c15f4f43c0add9f5d5f2f4a501be387e"}
Nov 03 13:43:13.769343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.769314    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:13.854602 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.854568    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p\" (UniqueName: \"kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p\") pod \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") "
Nov 03 13:43:13.854842 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.854637    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca\") pod \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") "
Nov 03 13:43:13.854842 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.854678    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca\") pod \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\" (UID: \"4756ee1e-98ec-4841-a9e1-dd6b04383dab\") "
Nov 03 13:43:13.855082 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.855058    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca" pod "4756ee1e-98ec-4841-a9e1-dd6b04383dab" (UID: "4756ee1e-98ec-4841-a9e1-dd6b04383dab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:13.857036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.857008    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca" pod "4756ee1e-98ec-4841-a9e1-dd6b04383dab" (UID: "4756ee1e-98ec-4841-a9e1-dd6b04383dab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:13.857147 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.857045    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p" pod "4756ee1e-98ec-4841-a9e1-dd6b04383dab" (UID: "4756ee1e-98ec-4841-a9e1-dd6b04383dab"). InnerVolumeSpecName "kube-api-access-rnk2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:13.956645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.956553    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:13.956645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.956586    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4756ee1e-98ec-4841-a9e1-dd6b04383dab-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:13.956645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:13.956595    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnk2p\" (UniqueName: \"kubernetes.io/projected/4756ee1e-98ec-4841-a9e1-dd6b04383dab-kube-api-access-rnk2p\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.631849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:14.631814    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"
Nov 03 13:43:14.631849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:14.631841    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp" event={"ID":"4756ee1e-98ec-4841-a9e1-dd6b04383dab","Type":"ContainerDied","Data":"884edc9e9157660d151935f6bd8d207562fa6fe16ce6e67f2fd1bc25c7c7a06a"}
Nov 03 13:43:14.632092 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:14.631891    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="884edc9e9157660d151935f6bd8d207562fa6fe16ce6e67f2fd1bc25c7c7a06a"
Nov 03 13:43:21.668722 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.668687    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"]
Nov 03 13:43:21.669170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.669099    4546 cpu_manager.go:414] "containerMap: removing container" podUID="4756ee1e-98ec-4841-a9e1-dd6b04383dab" containerName="revoke-keys"
Nov 03 13:43:21.669170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.669111    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4756ee1e-98ec-4841-a9e1-dd6b04383dab" containerName="revoke-keys"
Nov 03 13:43:21.669254 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.669186    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="4756ee1e-98ec-4841-a9e1-dd6b04383dab" containerName="revoke-keys"
Nov 03 13:43:21.672424 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.672400    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.721319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.721274    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:43:21.721319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.721291    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:43:21.721319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.721310    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:43:21.733992 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.733958    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.734167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.734085    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.734167 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.734157    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ggf\" (UniqueName: \"kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.767460 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.767427    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:43:21.769432 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.769410    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"]
Nov 03 13:43:21.834988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.834944    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ggf\" (UniqueName: \"kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.835190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.835066    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.835190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.835109    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.835872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.835853    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.837658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.837640    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.890070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.890034    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ggf\" (UniqueName: \"kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf\") pod \"maas-api-revoke-sub-f43a23ab-58vx7\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:21.985143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:21.985104    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:22.353860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:22.353831    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"]
Nov 03 13:43:22.355523 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:43:22.355492    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b9e71e_cf93_428e_93c5_12ad553b3e91.slice/crio-9f0a94b8d294edb259f88677e4348fae8966562dda80b729575340ef8c6e27fd WatchSource:0}: Error finding container 9f0a94b8d294edb259f88677e4348fae8966562dda80b729575340ef8c6e27fd: Status 404 returned error can't find the container with id 9f0a94b8d294edb259f88677e4348fae8966562dda80b729575340ef8c6e27fd
Nov 03 13:43:22.671689 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:22.671650    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7" event={"ID":"90b9e71e-cf93-428e-93c5-12ad553b3e91","Type":"ContainerStarted","Data":"d8e7a6603c7ce1a811c1118a2c293976f9e119bd5aebb233f7ddf3501a5a2f28"}
Nov 03 13:43:22.672120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:22.671714    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7" event={"ID":"90b9e71e-cf93-428e-93c5-12ad553b3e91","Type":"ContainerStarted","Data":"9f0a94b8d294edb259f88677e4348fae8966562dda80b729575340ef8c6e27fd"}
Nov 03 13:43:22.778283 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:22.778232    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7" podStartSLOduration=1.778215952 podStartE2EDuration="1.778215952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:22.777851039 +0000 UTC m=+6744.349434978" watchObservedRunningTime="2026-11-03 13:43:22.778215952 +0000 UTC m=+6744.349799861"
Nov 03 13:43:23.678071 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:23.678037    4546 generic.go:356] "Generic (PLEG): container finished" podID="90b9e71e-cf93-428e-93c5-12ad553b3e91" containerID="d8e7a6603c7ce1a811c1118a2c293976f9e119bd5aebb233f7ddf3501a5a2f28" exitCode=0
Nov 03 13:43:23.678523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:23.678125    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7" event={"ID":"90b9e71e-cf93-428e-93c5-12ad553b3e91","Type":"ContainerDied","Data":"d8e7a6603c7ce1a811c1118a2c293976f9e119bd5aebb233f7ddf3501a5a2f28"}
Nov 03 13:43:24.823509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.823481    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:24.869694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.869649    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf\" (UniqueName: \"kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf\") pod \"90b9e71e-cf93-428e-93c5-12ad553b3e91\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") "
Nov 03 13:43:24.869927 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.869713    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca\") pod \"90b9e71e-cf93-428e-93c5-12ad553b3e91\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") "
Nov 03 13:43:24.869927 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.869869    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca\") pod \"90b9e71e-cf93-428e-93c5-12ad553b3e91\" (UID: \"90b9e71e-cf93-428e-93c5-12ad553b3e91\") "
Nov 03 13:43:24.870102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.870076    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca" pod "90b9e71e-cf93-428e-93c5-12ad553b3e91" (UID: "90b9e71e-cf93-428e-93c5-12ad553b3e91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:24.870234 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.870218    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:24.872168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.872130    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf" pod "90b9e71e-cf93-428e-93c5-12ad553b3e91" (UID: "90b9e71e-cf93-428e-93c5-12ad553b3e91"). InnerVolumeSpecName "kube-api-access-n6ggf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:24.872168 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.872149    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca" pod "90b9e71e-cf93-428e-93c5-12ad553b3e91" (UID: "90b9e71e-cf93-428e-93c5-12ad553b3e91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:24.971481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.971444    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e71e-cf93-428e-93c5-12ad553b3e91-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:24.971481 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:24.971474    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ggf\" (UniqueName: \"kubernetes.io/projected/90b9e71e-cf93-428e-93c5-12ad553b3e91-kube-api-access-n6ggf\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:25.692335 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:25.692293    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7" event={"ID":"90b9e71e-cf93-428e-93c5-12ad553b3e91","Type":"ContainerDied","Data":"9f0a94b8d294edb259f88677e4348fae8966562dda80b729575340ef8c6e27fd"}
Nov 03 13:43:25.692534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:25.692380    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0a94b8d294edb259f88677e4348fae8966562dda80b729575340ef8c6e27fd"
Nov 03 13:43:25.692534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:25.692308    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"
Nov 03 13:43:37.652136 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:43:37.652096    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:43:55.051176 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.051140    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"]
Nov 03 13:43:55.051594 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.051543    4546 cpu_manager.go:414] "containerMap: removing container" podUID="90b9e71e-cf93-428e-93c5-12ad553b3e91" containerName="revoke-keys"
Nov 03 13:43:55.051594 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.051554    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b9e71e-cf93-428e-93c5-12ad553b3e91" containerName="revoke-keys"
Nov 03 13:43:55.051676 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.051624    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b9e71e-cf93-428e-93c5-12ad553b3e91" containerName="revoke-keys"
Nov 03 13:43:55.054810 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.054786    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.097703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.097655    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.097934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.097847    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.097934 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.097891    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8gt\" (UniqueName: \"kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.103557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.103536    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:43:55.149834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.149789    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:43:55.150036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.149930    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:43:55.150036 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.150026    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:43:55.151966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.151945    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"]
Nov 03 13:43:55.198503 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.198467    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.198709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.198519    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8gt\" (UniqueName: \"kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.198709 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.198572    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.199265 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.199233    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.201173 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.201147    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.252394 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.252356    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8gt\" (UniqueName: \"kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt\") pod \"maas-api-revoke-sub-6414fe74-rncmf\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.372399 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.372299    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:55.734864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.734817    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"]
Nov 03 13:43:55.736921 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:43:55.736890    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03198087_5a86_405f_936f_29c9787410cb.slice/crio-32c9017f6b7ceaa95ee8328bc34f3ed74816368e2a60622291b66a3ce76d7945 WatchSource:0}: Error finding container 32c9017f6b7ceaa95ee8328bc34f3ed74816368e2a60622291b66a3ce76d7945: Status 404 returned error can't find the container with id 32c9017f6b7ceaa95ee8328bc34f3ed74816368e2a60622291b66a3ce76d7945
Nov 03 13:43:55.853247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:55.853214    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf" event={"ID":"03198087-5a86-405f-936f-29c9787410cb","Type":"ContainerStarted","Data":"32c9017f6b7ceaa95ee8328bc34f3ed74816368e2a60622291b66a3ce76d7945"}
Nov 03 13:43:56.859708 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:56.859613    4546 generic.go:356] "Generic (PLEG): container finished" podID="03198087-5a86-405f-936f-29c9787410cb" containerID="81457cc8ad5464e4d7db59289369d5aea9cfa291e38ec9fcf3964eac358e642f" exitCode=0
Nov 03 13:43:56.859708 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:56.859695    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf" event={"ID":"03198087-5a86-405f-936f-29c9787410cb","Type":"ContainerDied","Data":"81457cc8ad5464e4d7db59289369d5aea9cfa291e38ec9fcf3964eac358e642f"}
Nov 03 13:43:58.001740 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.001712    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:58.045359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.045318    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"]
Nov 03 13:43:58.051701 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.051670    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52c211b-jwq5q"]
Nov 03 13:43:58.126525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.126434    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca\") pod \"03198087-5a86-405f-936f-29c9787410cb\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") "
Nov 03 13:43:58.126525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.126491    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca\") pod \"03198087-5a86-405f-936f-29c9787410cb\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") "
Nov 03 13:43:58.126733 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.126649    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt\" (UniqueName: \"kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt\") pod \"03198087-5a86-405f-936f-29c9787410cb\" (UID: \"03198087-5a86-405f-936f-29c9787410cb\") "
Nov 03 13:43:58.126894 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.126845    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca" pod "03198087-5a86-405f-936f-29c9787410cb" (UID: "03198087-5a86-405f-936f-29c9787410cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:58.127049 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.127031    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03198087-5a86-405f-936f-29c9787410cb-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:58.128844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.128820    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca" pod "03198087-5a86-405f-936f-29c9787410cb" (UID: "03198087-5a86-405f-936f-29c9787410cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:58.128985 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.128931    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt" pod "03198087-5a86-405f-936f-29c9787410cb" (UID: "03198087-5a86-405f-936f-29c9787410cb"). InnerVolumeSpecName "kube-api-access-ph8gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:58.228140 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.228101    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03198087-5a86-405f-936f-29c9787410cb-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:58.228140 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.228135    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8gt\" (UniqueName: \"kubernetes.io/projected/03198087-5a86-405f-936f-29c9787410cb-kube-api-access-ph8gt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:58.572569 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.572533    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c854e033-3797-4cb6-888b-f9b320db95ef" path="/var/lib/kubelet/pods/c854e033-3797-4cb6-888b-f9b320db95ef/volumes"
Nov 03 13:43:58.871012 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.870921    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"
Nov 03 13:43:58.871012 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.870959    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf" event={"ID":"03198087-5a86-405f-936f-29c9787410cb","Type":"ContainerDied","Data":"32c9017f6b7ceaa95ee8328bc34f3ed74816368e2a60622291b66a3ce76d7945"}
Nov 03 13:43:58.871012 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:43:58.871007    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c9017f6b7ceaa95ee8328bc34f3ed74816368e2a60622291b66a3ce76d7945"
Nov 03 13:44:03.573645 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:03.573611    4546 scope.go:122] "RemoveContainer" containerID="b82f1d8d781783ce442638a835ec5787cb5749a4f8df373e3d5dd3f600585e5f"
Nov 03 13:44:07.650571 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:44:07.650486    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:44:37.532216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.532182    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"]
Nov 03 13:44:37.532824 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.532803    4546 cpu_manager.go:414] "containerMap: removing container" podUID="03198087-5a86-405f-936f-29c9787410cb" containerName="revoke-keys"
Nov 03 13:44:37.532876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.532827    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03198087-5a86-405f-936f-29c9787410cb" containerName="revoke-keys"
Nov 03 13:44:37.532962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.532942    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="03198087-5a86-405f-936f-29c9787410cb" containerName="revoke-keys"
Nov 03 13:44:37.536190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.536168    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.631359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.631326    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:44:37.631359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.631351    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:44:37.631359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.631351    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:44:37.631359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.631322    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:44:37.633401 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.633368    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"]
Nov 03 13:44:37.640948 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.640902    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.641107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.640986    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdq9\" (UniqueName: \"kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.641224 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.641201    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.652348 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:44:37.652313    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:44:37.741966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.741921    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.742162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.741975    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.742162 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.742020    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdq9\" (UniqueName: \"kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.742591 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.742570    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.744666 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.744647    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.796795 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.796691    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdq9\" (UniqueName: \"kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9\") pod \"maas-api-revoke-sub-b691b413-8f6n5\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:37.853715 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:37.853674    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:38.219390 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:38.219365    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"]
Nov 03 13:44:38.222196 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:44:38.222161    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcec510_4066_463a_8700_b2a961d386d9.slice/crio-143832ccbf5195b3a57291440b3ff7136333fd1d1d057e2e5dc466e24a655069 WatchSource:0}: Error finding container 143832ccbf5195b3a57291440b3ff7136333fd1d1d057e2e5dc466e24a655069: Status 404 returned error can't find the container with id 143832ccbf5195b3a57291440b3ff7136333fd1d1d057e2e5dc466e24a655069
Nov 03 13:44:39.078264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:39.078238    4546 generic.go:356] "Generic (PLEG): container finished" podID="5dcec510-4066-463a-8700-b2a961d386d9" containerID="79d24052395e989c1acfd7bdc1b3e1104ccd4652930aeef96907becf4ce5c981" exitCode=0
Nov 03 13:44:39.078671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:39.078323    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5" event={"ID":"5dcec510-4066-463a-8700-b2a961d386d9","Type":"ContainerDied","Data":"79d24052395e989c1acfd7bdc1b3e1104ccd4652930aeef96907becf4ce5c981"}
Nov 03 13:44:39.078671 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:39.078391    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5" event={"ID":"5dcec510-4066-463a-8700-b2a961d386d9","Type":"ContainerStarted","Data":"143832ccbf5195b3a57291440b3ff7136333fd1d1d057e2e5dc466e24a655069"}
Nov 03 13:44:40.221224 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.221200    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:40.371189 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.371083    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9\" (UniqueName: \"kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9\") pod \"5dcec510-4066-463a-8700-b2a961d386d9\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") "
Nov 03 13:44:40.371189 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.371138    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca\") pod \"5dcec510-4066-463a-8700-b2a961d386d9\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") "
Nov 03 13:44:40.371189 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.371159    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca\") pod \"5dcec510-4066-463a-8700-b2a961d386d9\" (UID: \"5dcec510-4066-463a-8700-b2a961d386d9\") "
Nov 03 13:44:40.371599 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.371565    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca" pod "5dcec510-4066-463a-8700-b2a961d386d9" (UID: "5dcec510-4066-463a-8700-b2a961d386d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:40.373536 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.373505    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca" pod "5dcec510-4066-463a-8700-b2a961d386d9" (UID: "5dcec510-4066-463a-8700-b2a961d386d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:40.373685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.373624    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9" pod "5dcec510-4066-463a-8700-b2a961d386d9" (UID: "5dcec510-4066-463a-8700-b2a961d386d9"). InnerVolumeSpecName "kube-api-access-ttdq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:40.472274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.472230    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdq9\" (UniqueName: \"kubernetes.io/projected/5dcec510-4066-463a-8700-b2a961d386d9-kube-api-access-ttdq9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:40.472274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.472268    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:40.472274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:40.472282    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcec510-4066-463a-8700-b2a961d386d9-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:41.090386 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:41.090354    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"
Nov 03 13:44:41.090580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:41.090389    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5" event={"ID":"5dcec510-4066-463a-8700-b2a961d386d9","Type":"ContainerDied","Data":"143832ccbf5195b3a57291440b3ff7136333fd1d1d057e2e5dc466e24a655069"}
Nov 03 13:44:41.090580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:41.090437    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143832ccbf5195b3a57291440b3ff7136333fd1d1d057e2e5dc466e24a655069"
Nov 03 13:44:48.314031 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.314000    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"]
Nov 03 13:44:48.314572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.314415    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5dcec510-4066-463a-8700-b2a961d386d9" containerName="revoke-keys"
Nov 03 13:44:48.314572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.314428    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcec510-4066-463a-8700-b2a961d386d9" containerName="revoke-keys"
Nov 03 13:44:48.314572 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.314518    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcec510-4066-463a-8700-b2a961d386d9" containerName="revoke-keys"
Nov 03 13:44:48.318010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.317991    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.412547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.412511    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:44:48.412547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.412548    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:44:48.412800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.412550    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:44:48.414619 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.414599    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"]
Nov 03 13:44:48.448327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.448289    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.448534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.448407    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4l7\" (UniqueName: \"kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.448534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.448453    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.466990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.466957    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:44:48.549241 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.549201    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.549469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.549257    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.549469 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.549333    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4l7\" (UniqueName: \"kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.549912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.549889    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.551867 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.551847    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.602962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.602880    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4l7\" (UniqueName: \"kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7\") pod \"maas-api-revoke-sub-48061e9f-xs8pq\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.632682 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.632647    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:48.995522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:48.995460    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"]
Nov 03 13:44:48.998005 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:44:48.997973    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01463f96_983a_4f86_b409_c91e1e09e51e.slice/crio-4d9cf5a7d8311757d0e86ca70a7fd68ca87f6bf91b45a5a664ecccb439811ef7 WatchSource:0}: Error finding container 4d9cf5a7d8311757d0e86ca70a7fd68ca87f6bf91b45a5a664ecccb439811ef7: Status 404 returned error can't find the container with id 4d9cf5a7d8311757d0e86ca70a7fd68ca87f6bf91b45a5a664ecccb439811ef7
Nov 03 13:44:49.128975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:49.128928    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq" event={"ID":"01463f96-983a-4f86-b409-c91e1e09e51e","Type":"ContainerStarted","Data":"4d9cf5a7d8311757d0e86ca70a7fd68ca87f6bf91b45a5a664ecccb439811ef7"}
Nov 03 13:44:50.135258 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:50.135164    4546 generic.go:356] "Generic (PLEG): container finished" podID="01463f96-983a-4f86-b409-c91e1e09e51e" containerID="7f0c8d7d9a28c28537c8eab63d0a696bbb5ea2e72d588acf2fa3a21802efc27c" exitCode=0
Nov 03 13:44:50.135258 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:50.135203    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq" event={"ID":"01463f96-983a-4f86-b409-c91e1e09e51e","Type":"ContainerDied","Data":"7f0c8d7d9a28c28537c8eab63d0a696bbb5ea2e72d588acf2fa3a21802efc27c"}
Nov 03 13:44:51.282512 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.282486    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:51.376050 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.376017    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca\") pod \"01463f96-983a-4f86-b409-c91e1e09e51e\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") "
Nov 03 13:44:51.376247 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.376098    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca\") pod \"01463f96-983a-4f86-b409-c91e1e09e51e\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") "
Nov 03 13:44:51.376302 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.376260    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7\" (UniqueName: \"kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7\") pod \"01463f96-983a-4f86-b409-c91e1e09e51e\" (UID: \"01463f96-983a-4f86-b409-c91e1e09e51e\") "
Nov 03 13:44:51.376485 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.376463    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca" pod "01463f96-983a-4f86-b409-c91e1e09e51e" (UID: "01463f96-983a-4f86-b409-c91e1e09e51e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:51.376606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.376588    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:51.378480 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.378457    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca" pod "01463f96-983a-4f86-b409-c91e1e09e51e" (UID: "01463f96-983a-4f86-b409-c91e1e09e51e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:51.378544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.378507    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7" pod "01463f96-983a-4f86-b409-c91e1e09e51e" (UID: "01463f96-983a-4f86-b409-c91e1e09e51e"). InnerVolumeSpecName "kube-api-access-lr4l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:51.477831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.477788    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr4l7\" (UniqueName: \"kubernetes.io/projected/01463f96-983a-4f86-b409-c91e1e09e51e-kube-api-access-lr4l7\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:51.477831 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:51.477827    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01463f96-983a-4f86-b409-c91e1e09e51e-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:52.145918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:52.145871    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq" event={"ID":"01463f96-983a-4f86-b409-c91e1e09e51e","Type":"ContainerDied","Data":"4d9cf5a7d8311757d0e86ca70a7fd68ca87f6bf91b45a5a664ecccb439811ef7"}
Nov 03 13:44:52.145918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:52.145915    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"
Nov 03 13:44:52.146211 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:44:52.145922    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9cf5a7d8311757d0e86ca70a7fd68ca87f6bf91b45a5a664ecccb439811ef7"
Nov 03 13:45:00.162443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.162402    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"]
Nov 03 13:45:00.163054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.163034    4546 cpu_manager.go:414] "containerMap: removing container" podUID="01463f96-983a-4f86-b409-c91e1e09e51e" containerName="revoke-keys"
Nov 03 13:45:00.163105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.163058    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01463f96-983a-4f86-b409-c91e1e09e51e" containerName="revoke-keys"
Nov 03 13:45:00.163160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.163148    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="01463f96-983a-4f86-b409-c91e1e09e51e" containerName="revoke-keys"
Nov 03 13:45:00.167558 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.167535    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.262243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.262207    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-fzqp6\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.262454 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.262278    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8qx\" (UniqueName: \"kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx\") pod \"maas-api-key-cleanup-29895225-fzqp6\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.262856 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.262832    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:45:00.262941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.262895    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:45:00.262941 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.262908    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:45:00.265158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.265136    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"]
Nov 03 13:45:00.363218 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.363182    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8qx\" (UniqueName: \"kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx\") pod \"maas-api-key-cleanup-29895225-fzqp6\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.363427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.363299    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-fzqp6\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.363972 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.363951    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-fzqp6\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.417040 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.416963    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8qx\" (UniqueName: \"kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx\") pod \"maas-api-key-cleanup-29895225-fzqp6\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.480885 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.480852    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:00.844065 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:00.844041    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"]
Nov 03 13:45:00.845615 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:45:00.845583    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c228c69_eb29_4483_87cf_4c957c50a32c.slice/crio-e5d3d4d459d36e95347d5a3d0b31ef8653dbf872fca472f83f6581c59fa740e4 WatchSource:0}: Error finding container e5d3d4d459d36e95347d5a3d0b31ef8653dbf872fca472f83f6581c59fa740e4: Status 404 returned error can't find the container with id e5d3d4d459d36e95347d5a3d0b31ef8653dbf872fca472f83f6581c59fa740e4
Nov 03 13:45:01.191594 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:01.191550    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerStarted","Data":"2c3e45d2ee61481c28eecb397be00b6a4bc3023a153d0193985e5dcbcb0e290f"}
Nov 03 13:45:01.192039 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:01.191626    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerStarted","Data":"e5d3d4d459d36e95347d5a3d0b31ef8653dbf872fca472f83f6581c59fa740e4"}
Nov 03 13:45:01.300130 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:01.300074    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" podStartSLOduration=1.300060499 podStartE2EDuration="1.300060499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.298687687 +0000 UTC m=+6842.870271619" watchObservedRunningTime="2026-11-03 13:45:01.300060499 +0000 UTC m=+6842.871644410"
Nov 03 13:45:02.198429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:02.198334    4546 generic.go:356] "Generic (PLEG): container finished" podID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerID="2c3e45d2ee61481c28eecb397be00b6a4bc3023a153d0193985e5dcbcb0e290f" exitCode=60
Nov 03 13:45:02.198930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:02.198418    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerDied","Data":"2c3e45d2ee61481c28eecb397be00b6a4bc3023a153d0193985e5dcbcb0e290f"}
Nov 03 13:45:02.198930 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:02.198682    4546 scope.go:122] "RemoveContainer" containerID="2c3e45d2ee61481c28eecb397be00b6a4bc3023a153d0193985e5dcbcb0e290f"
Nov 03 13:45:02.682196 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:02.682152    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:02.682385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:02.682224    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:03.204703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:03.204665    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerStarted","Data":"8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4"}
Nov 03 13:45:03.597251 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:03.597219    4546 scope.go:122] "RemoveContainer" containerID="66da3b5be9e0c2399527b0fbe55d45410b88737718e12707d28e98e4547cca3e"
Nov 03 13:45:04.210832 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:04.210794    4546 generic.go:356] "Generic (PLEG): container finished" podID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerID="8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4" exitCode=60
Nov 03 13:45:04.211354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:04.210891    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerDied","Data":"8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4"}
Nov 03 13:45:04.211354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:04.210965    4546 scope.go:122] "RemoveContainer" containerID="2c3e45d2ee61481c28eecb397be00b6a4bc3023a153d0193985e5dcbcb0e290f"
Nov 03 13:45:04.211354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:04.211111    4546 scope.go:122] "RemoveContainer" containerID="8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4"
Nov 03 13:45:04.211521 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:45:04.211361    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-fzqp6_odh-ai-gateway-infra(7c228c69-eb29-4483-87cf-4c957c50a32c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c"
Nov 03 13:45:05.217646 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:05.217615    4546 scope.go:122] "RemoveContainer" containerID="8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4"
Nov 03 13:45:05.218099 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:45:05.217874    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-fzqp6_odh-ai-gateway-infra(7c228c69-eb29-4483-87cf-4c957c50a32c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c"
Nov 03 13:45:07.656342 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:45:07.656304    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:45:16.585778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:16.582650    4546 scope.go:122] "RemoveContainer" containerID="8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4"
Nov 03 13:45:17.353775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:17.349081    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerStarted","Data":"ff510a8837200716ed36a733260c4057f3a48bcb9de3227a4bc1c8a6a81215b5"}
Nov 03 13:45:17.692055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:17.691948    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:17.692636 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:17.692604    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:18.355668 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:18.355638    4546 generic.go:356] "Generic (PLEG): container finished" podID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerID="ff510a8837200716ed36a733260c4057f3a48bcb9de3227a4bc1c8a6a81215b5" exitCode=60
Nov 03 13:45:18.355890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:18.355709    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerDied","Data":"ff510a8837200716ed36a733260c4057f3a48bcb9de3227a4bc1c8a6a81215b5"}
Nov 03 13:45:18.355890 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:18.355790    4546 scope.go:122] "RemoveContainer" containerID="8a624665556b9d237ea9392433c9e910327933a36a49122ee8acf0c86c4e98b4"
Nov 03 13:45:18.356172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:18.356154    4546 scope.go:122] "RemoveContainer" containerID="ff510a8837200716ed36a733260c4057f3a48bcb9de3227a4bc1c8a6a81215b5"
Nov 03 13:45:18.356432 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:45:18.356410    4546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-fzqp6_odh-ai-gateway-infra(7c228c69-eb29-4483-87cf-4c957c50a32c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c"
Nov 03 13:45:18.482316 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:18.482268    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"]
Nov 03 13:45:19.506914 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.506890    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:19.611905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.611861    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca\") pod \"7c228c69-eb29-4483-87cf-4c957c50a32c\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") "
Nov 03 13:45:19.612102 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.611954    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx\" (UniqueName: \"kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx\") pod \"7c228c69-eb29-4483-87cf-4c957c50a32c\" (UID: \"7c228c69-eb29-4483-87cf-4c957c50a32c\") "
Nov 03 13:45:19.612309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.612255    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca" pod "7c228c69-eb29-4483-87cf-4c957c50a32c" (UID: "7c228c69-eb29-4483-87cf-4c957c50a32c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:19.614592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.614562    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx" pod "7c228c69-eb29-4483-87cf-4c957c50a32c" (UID: "7c228c69-eb29-4483-87cf-4c957c50a32c"). InnerVolumeSpecName "kube-api-access-tb8qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:19.713261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.713221    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c228c69-eb29-4483-87cf-4c957c50a32c-maas-api-service-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.713261 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:19.713255    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb8qx\" (UniqueName: \"kubernetes.io/projected/7c228c69-eb29-4483-87cf-4c957c50a32c-kube-api-access-tb8qx\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.368071 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:20.368027    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6" event={"ID":"7c228c69-eb29-4483-87cf-4c957c50a32c","Type":"ContainerDied","Data":"e5d3d4d459d36e95347d5a3d0b31ef8653dbf872fca472f83f6581c59fa740e4"}
Nov 03 13:45:20.368071 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:20.368059    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"
Nov 03 13:45:20.368373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:20.368097    4546 scope.go:122] "RemoveContainer" containerID="ff510a8837200716ed36a733260c4057f3a48bcb9de3227a4bc1c8a6a81215b5"
Nov 03 13:45:20.521427 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:20.521388    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"]
Nov 03 13:45:20.572327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:20.572299    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-fzqp6"]
Nov 03 13:45:22.571769 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:22.571709    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" path="/var/lib/kubelet/pods/7c228c69-eb29-4483-87cf-4c957c50a32c/volumes"
Nov 03 13:45:26.051514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:26.051480    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"]
Nov 03 13:45:26.099075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:26.099037    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vm2pj"]
Nov 03 13:45:26.574181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:26.574141    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ebb8fa6-3c8f-43a9-b365-560bcb357d30" path="/var/lib/kubelet/pods/3ebb8fa6-3c8f-43a9-b365-560bcb357d30/volumes"
Nov 03 13:45:36.343223 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343185    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"]
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343561    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343572    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343590    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343595    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343664    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343674    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.343725 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.343680    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:36.346995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.346972    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.388197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.388164    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.388197 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.388200    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.388418 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.388324    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lpt\" (UniqueName: \"kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.397002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.396974    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:45:36.443112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.443081    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:45:36.443112 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.443094    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:45:36.443379 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.443118    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:45:36.445434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.445411    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"]
Nov 03 13:45:36.489545 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.489513    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.489719 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.489549    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.489719 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.489627    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lpt\" (UniqueName: \"kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.490324 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.490295    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.492183 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.492161    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.545464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.545428    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lpt\" (UniqueName: \"kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt\") pod \"maas-api-revoke-sub-6ec59d4a-psd4r\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:36.659520 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:36.659424    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:37.031543 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:37.031517    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"]
Nov 03 13:45:37.033529 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:45:37.033500    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819e7513_46bc_4f61_b929_85e7027d777c.slice/crio-02b86850266b262172b99f00750d615cc185347ea060a17b3a10162d0c0a458d WatchSource:0}: Error finding container 02b86850266b262172b99f00750d615cc185347ea060a17b3a10162d0c0a458d: Status 404 returned error can't find the container with id 02b86850266b262172b99f00750d615cc185347ea060a17b3a10162d0c0a458d
Nov 03 13:45:37.452623 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:37.452587    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r" event={"ID":"819e7513-46bc-4f61-b929-85e7027d777c","Type":"ContainerStarted","Data":"020aab405b0ba6bc8a09eece8ac4ac8aa1de35a422f138b1e99f628c81b8b232"}
Nov 03 13:45:37.453107 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:37.452644    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r" event={"ID":"819e7513-46bc-4f61-b929-85e7027d777c","Type":"ContainerStarted","Data":"02b86850266b262172b99f00750d615cc185347ea060a17b3a10162d0c0a458d"}
Nov 03 13:45:37.565386 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:37.565278    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r" podStartSLOduration=1.565262567 podStartE2EDuration="1.565262567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:37.563048899 +0000 UTC m=+6879.134632793" watchObservedRunningTime="2026-11-03 13:45:37.565262567 +0000 UTC m=+6879.136846479"
Nov 03 13:45:37.651148 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:45:37.651072    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:45:38.458633 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:38.458598    4546 generic.go:356] "Generic (PLEG): container finished" podID="819e7513-46bc-4f61-b929-85e7027d777c" containerID="020aab405b0ba6bc8a09eece8ac4ac8aa1de35a422f138b1e99f628c81b8b232" exitCode=0
Nov 03 13:45:38.459081 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:38.458682    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r" event={"ID":"819e7513-46bc-4f61-b929-85e7027d777c","Type":"ContainerDied","Data":"020aab405b0ba6bc8a09eece8ac4ac8aa1de35a422f138b1e99f628c81b8b232"}
Nov 03 13:45:39.603652 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.603624    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:39.724009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.723925    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt\" (UniqueName: \"kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt\") pod \"819e7513-46bc-4f61-b929-85e7027d777c\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") "
Nov 03 13:45:39.724009 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.723973    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca\") pod \"819e7513-46bc-4f61-b929-85e7027d777c\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") "
Nov 03 13:45:39.724212 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.724059    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca\") pod \"819e7513-46bc-4f61-b929-85e7027d777c\" (UID: \"819e7513-46bc-4f61-b929-85e7027d777c\") "
Nov 03 13:45:39.724366 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.724344    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca" pod "819e7513-46bc-4f61-b929-85e7027d777c" (UID: "819e7513-46bc-4f61-b929-85e7027d777c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:39.724514 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.724491    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:39.726343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.726310    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca" pod "819e7513-46bc-4f61-b929-85e7027d777c" (UID: "819e7513-46bc-4f61-b929-85e7027d777c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:39.726440 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.726384    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt" pod "819e7513-46bc-4f61-b929-85e7027d777c" (UID: "819e7513-46bc-4f61-b929-85e7027d777c"). InnerVolumeSpecName "kube-api-access-69lpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:39.825739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.825705    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69lpt\" (UniqueName: \"kubernetes.io/projected/819e7513-46bc-4f61-b929-85e7027d777c-kube-api-access-69lpt\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:39.825739 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:39.825737    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819e7513-46bc-4f61-b929-85e7027d777c-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:40.469443 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:40.469405    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r" event={"ID":"819e7513-46bc-4f61-b929-85e7027d777c","Type":"ContainerDied","Data":"02b86850266b262172b99f00750d615cc185347ea060a17b3a10162d0c0a458d"}
Nov 03 13:45:40.469669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:40.469467    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02b86850266b262172b99f00750d615cc185347ea060a17b3a10162d0c0a458d"
Nov 03 13:45:40.469669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:40.469412    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec59d4a-psd4r"
Nov 03 13:45:47.104196 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.104159    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"]
Nov 03 13:45:47.104606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.104552    4546 cpu_manager.go:414] "containerMap: removing container" podUID="819e7513-46bc-4f61-b929-85e7027d777c" containerName="revoke-keys"
Nov 03 13:45:47.104606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.104564    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819e7513-46bc-4f61-b929-85e7027d777c" containerName="revoke-keys"
Nov 03 13:45:47.104606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.104588    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:47.104606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.104593    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c228c69-eb29-4483-87cf-4c957c50a32c" containerName="cleanup"
Nov 03 13:45:47.104803 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.104664    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="819e7513-46bc-4f61-b929-85e7027d777c" containerName="revoke-keys"
Nov 03 13:45:47.108236 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.108213    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.160511 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.160473    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"]
Nov 03 13:45:47.195339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.195301    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrj4\" (UniqueName: \"kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.195497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.195357    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.195497 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.195447    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.204020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.203983    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:45:47.204020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.203984    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:45:47.204020 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.204007    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:45:47.204242 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.204083    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:45:47.296698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.296656    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrj4\" (UniqueName: \"kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.296951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.296708    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.296951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.296741    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.297467 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.297442    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.299580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.299557    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.350775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.350727    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrj4\" (UniqueName: \"kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4\") pod \"maas-api-revoke-sub-9b489d3c-4tcqj\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.424702 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.424593    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:47.791727 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:47.791698    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"]
Nov 03 13:45:47.794000 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:45:47.793962    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda199a9ac_d070_4ca7_a03f_31e45513d2c0.slice/crio-05ba4805f7ed318d509e5fe4b4417d887c5eb2f30a2d38af59b91489f9db92bf WatchSource:0}: Error finding container 05ba4805f7ed318d509e5fe4b4417d887c5eb2f30a2d38af59b91489f9db92bf: Status 404 returned error can't find the container with id 05ba4805f7ed318d509e5fe4b4417d887c5eb2f30a2d38af59b91489f9db92bf
Nov 03 13:45:48.510614 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:48.510573    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj" event={"ID":"a199a9ac-d070-4ca7-a03f-31e45513d2c0","Type":"ContainerStarted","Data":"6ecbd2eae523714826eba7cfb899c4e0cdc88042acf30ca53854e9ee429bd753"}
Nov 03 13:45:48.511114 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:48.510640    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj" event={"ID":"a199a9ac-d070-4ca7-a03f-31e45513d2c0","Type":"ContainerStarted","Data":"05ba4805f7ed318d509e5fe4b4417d887c5eb2f30a2d38af59b91489f9db92bf"}
Nov 03 13:45:48.620057 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:48.620004    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj" podStartSLOduration=1.619989232 podStartE2EDuration="1.619989232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:48.617680928 +0000 UTC m=+6890.189264858" watchObservedRunningTime="2026-11-03 13:45:48.619989232 +0000 UTC m=+6890.191573167"
Nov 03 13:45:49.520190 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:49.520154    4546 generic.go:356] "Generic (PLEG): container finished" podID="a199a9ac-d070-4ca7-a03f-31e45513d2c0" containerID="6ecbd2eae523714826eba7cfb899c4e0cdc88042acf30ca53854e9ee429bd753" exitCode=0
Nov 03 13:45:49.520901 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:49.520249    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj" event={"ID":"a199a9ac-d070-4ca7-a03f-31e45513d2c0","Type":"ContainerDied","Data":"6ecbd2eae523714826eba7cfb899c4e0cdc88042acf30ca53854e9ee429bd753"}
Nov 03 13:45:50.992525 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:50.992493    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:45:51.086879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.086356    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca\") pod \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") "
Nov 03 13:45:51.087312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.087288    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4\" (UniqueName: \"kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4\") pod \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") "
Nov 03 13:45:51.087544 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.087527    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca\") pod \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\" (UID: \"a199a9ac-d070-4ca7-a03f-31e45513d2c0\") "
Nov 03 13:45:51.088339 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.088314    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca" pod "a199a9ac-d070-4ca7-a03f-31e45513d2c0" (UID: "a199a9ac-d070-4ca7-a03f-31e45513d2c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:51.088966 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.088926    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca" pod "a199a9ac-d070-4ca7-a03f-31e45513d2c0" (UID: "a199a9ac-d070-4ca7-a03f-31e45513d2c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:51.090984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.090963    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4" pod "a199a9ac-d070-4ca7-a03f-31e45513d2c0" (UID: "a199a9ac-d070-4ca7-a03f-31e45513d2c0"). InnerVolumeSpecName "kube-api-access-fdrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:51.189858 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.189729    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrj4\" (UniqueName: \"kubernetes.io/projected/a199a9ac-d070-4ca7-a03f-31e45513d2c0-kube-api-access-fdrj4\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:51.190116 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.190094    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:51.190240 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.190226    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a199a9ac-d070-4ca7-a03f-31e45513d2c0-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:51.571940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.571898    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj" event={"ID":"a199a9ac-d070-4ca7-a03f-31e45513d2c0","Type":"ContainerDied","Data":"05ba4805f7ed318d509e5fe4b4417d887c5eb2f30a2d38af59b91489f9db92bf"}
Nov 03 13:45:51.572238 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.572218    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05ba4805f7ed318d509e5fe4b4417d887c5eb2f30a2d38af59b91489f9db92bf"
Nov 03 13:45:51.572426 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:45:51.572410    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b489d3c-4tcqj"
Nov 03 13:46:03.617054 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:03.617016    4546 scope.go:122] "RemoveContainer" containerID="077ea542f8081ed928e8ac85c8f015145ac16c515026f1b4c61314bb053091d4"
Nov 03 13:46:07.650795 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:46:07.650737    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:46:32.689592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:32.684291    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:46:32.689592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:32.684394    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:46:35.445738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.445699    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"]
Nov 03 13:46:35.446217 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.446130    4546 cpu_manager.go:414] "containerMap: removing container" podUID="a199a9ac-d070-4ca7-a03f-31e45513d2c0" containerName="revoke-keys"
Nov 03 13:46:35.446217 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.446142    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a199a9ac-d070-4ca7-a03f-31e45513d2c0" containerName="revoke-keys"
Nov 03 13:46:35.446217 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.446201    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="a199a9ac-d070-4ca7-a03f-31e45513d2c0" containerName="revoke-keys"
Nov 03 13:46:35.449371 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.449336    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.498317 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.498284    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:46:35.515571 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.515527    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:46:35.519285 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.519251    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"]
Nov 03 13:46:35.521304 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.521278    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.521429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.521411    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4km\" (UniqueName: \"kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.521472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.521444    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.544503 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.544471    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:46:35.544681 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.544573    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:46:35.622895 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.622846    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4km\" (UniqueName: \"kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.622895 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.622904    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.623140 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.622966    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.623728 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.623650    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.625583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.625558    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.676248 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.676218    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4km\" (UniqueName: \"kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km\") pod \"maas-api-revoke-sub-55f36919-t9znd\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:35.764492 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:35.764451    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:36.129294 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:36.129268    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"]
Nov 03 13:46:36.131578 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:46:36.131553    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb8ff4e_b8ff_4b13_8720_da6cc4a8400e.slice/crio-8412456a748c69f3cdeaa5209520fa1a263114ebe7ed44ba85a8636f065fadc9 WatchSource:0}: Error finding container 8412456a748c69f3cdeaa5209520fa1a263114ebe7ed44ba85a8636f065fadc9: Status 404 returned error can't find the container with id 8412456a748c69f3cdeaa5209520fa1a263114ebe7ed44ba85a8636f065fadc9
Nov 03 13:46:36.865726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:36.865686    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd" event={"ID":"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e","Type":"ContainerStarted","Data":"823adefe55d05a353091ea9a01e343045ac76c88eb81899b04d41373c2dfdd80"}
Nov 03 13:46:36.866179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:36.865740    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd" event={"ID":"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e","Type":"ContainerStarted","Data":"8412456a748c69f3cdeaa5209520fa1a263114ebe7ed44ba85a8636f065fadc9"}
Nov 03 13:46:37.018965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:37.018919    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd" podStartSLOduration=2.018903366 podStartE2EDuration="2.018903366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:36.972265781 +0000 UTC m=+6938.543849707" watchObservedRunningTime="2026-11-03 13:46:37.018903366 +0000 UTC m=+6938.590487277"
Nov 03 13:46:37.651625 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:46:37.651590    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:46:37.872429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:37.872395    4546 generic.go:356] "Generic (PLEG): container finished" podID="ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" containerID="823adefe55d05a353091ea9a01e343045ac76c88eb81899b04d41373c2dfdd80" exitCode=0
Nov 03 13:46:37.872938 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:37.872476    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd" event={"ID":"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e","Type":"ContainerDied","Data":"823adefe55d05a353091ea9a01e343045ac76c88eb81899b04d41373c2dfdd80"}
Nov 03 13:46:39.017513 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.017485    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:39.057171 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.057133    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca\") pod \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") "
Nov 03 13:46:39.057355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.057249    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca\") pod \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") "
Nov 03 13:46:39.057355 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.057308    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km\" (UniqueName: \"kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km\") pod \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\" (UID: \"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e\") "
Nov 03 13:46:39.057602 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.057578    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca" pod "ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" (UID: "ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:46:39.059456 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.059431    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca" pod "ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" (UID: "ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:46:39.059609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.059587    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km" pod "ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" (UID: "ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e"). InnerVolumeSpecName "kube-api-access-dd4km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:39.159134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.159032    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:39.159134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.159087    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:39.159134 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.159103    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4km\" (UniqueName: \"kubernetes.io/projected/ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e-kube-api-access-dd4km\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:39.889844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.889812    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd"
Nov 03 13:46:39.890066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.889804    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f36919-t9znd" event={"ID":"ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e","Type":"ContainerDied","Data":"8412456a748c69f3cdeaa5209520fa1a263114ebe7ed44ba85a8636f065fadc9"}
Nov 03 13:46:39.890066 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:39.889963    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8412456a748c69f3cdeaa5209520fa1a263114ebe7ed44ba85a8636f065fadc9"
Nov 03 13:46:46.231335 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.231295    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"]
Nov 03 13:46:46.231923 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.231902    4546 cpu_manager.go:414] "containerMap: removing container" podUID="ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" containerName="revoke-keys"
Nov 03 13:46:46.232017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.231925    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" containerName="revoke-keys"
Nov 03 13:46:46.232080 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.232029    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e" containerName="revoke-keys"
Nov 03 13:46:46.235121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.235097    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.327738 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.327674    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.327974 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.327814    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xll\" (UniqueName: \"kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.327974 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.327883    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.332985 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.332939    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:46:46.332985 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.332974    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:46:46.333216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.333013    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:46:46.333216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.333050    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:46:46.335277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.335253    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"]
Nov 03 13:46:46.429485 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.429445    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.429703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.429498    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xll\" (UniqueName: \"kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.429703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.429555    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.430166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.430145    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.432085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.432061    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.483921 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.483845    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xll\" (UniqueName: \"kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll\") pod \"maas-api-revoke-sub-58d324b0-kmw6l\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.548663 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.548633    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:46.924782 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:46.924742    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"]
Nov 03 13:46:46.927098 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:46:46.927066    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511caeb6_5ade_4d40_bd25_8b8a6d6eea05.slice/crio-534ddd4bdba83d58502dc2a385bbc2693dcbf48d254d782ccc22ee4590fe820b WatchSource:0}: Error finding container 534ddd4bdba83d58502dc2a385bbc2693dcbf48d254d782ccc22ee4590fe820b: Status 404 returned error can't find the container with id 534ddd4bdba83d58502dc2a385bbc2693dcbf48d254d782ccc22ee4590fe820b
Nov 03 13:46:47.929361 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:47.929327    4546 generic.go:356] "Generic (PLEG): container finished" podID="511caeb6-5ade-4d40-bd25-8b8a6d6eea05" containerID="c55d48b20514863775b15c9acb3c817ea4875b7f798b35f6ffe1fb127b34e2df" exitCode=0
Nov 03 13:46:47.929816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:47.929423    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l" event={"ID":"511caeb6-5ade-4d40-bd25-8b8a6d6eea05","Type":"ContainerDied","Data":"c55d48b20514863775b15c9acb3c817ea4875b7f798b35f6ffe1fb127b34e2df"}
Nov 03 13:46:47.929816 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:47.929478    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l" event={"ID":"511caeb6-5ade-4d40-bd25-8b8a6d6eea05","Type":"ContainerStarted","Data":"534ddd4bdba83d58502dc2a385bbc2693dcbf48d254d782ccc22ee4590fe820b"}
Nov 03 13:46:49.068495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.068467    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:49.156521 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.156486    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca\") pod \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") "
Nov 03 13:46:49.156716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.156546    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll\" (UniqueName: \"kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll\") pod \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") "
Nov 03 13:46:49.156716 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.156577    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca\") pod \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\" (UID: \"511caeb6-5ade-4d40-bd25-8b8a6d6eea05\") "
Nov 03 13:46:49.157061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.157037    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca" pod "511caeb6-5ade-4d40-bd25-8b8a6d6eea05" (UID: "511caeb6-5ade-4d40-bd25-8b8a6d6eea05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:46:49.158995 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.158972    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca" pod "511caeb6-5ade-4d40-bd25-8b8a6d6eea05" (UID: "511caeb6-5ade-4d40-bd25-8b8a6d6eea05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:46:49.159106 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.158994    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll" pod "511caeb6-5ade-4d40-bd25-8b8a6d6eea05" (UID: "511caeb6-5ade-4d40-bd25-8b8a6d6eea05"). InnerVolumeSpecName "kube-api-access-v9xll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:49.258274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.258235    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:49.258274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.258266    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xll\" (UniqueName: \"kubernetes.io/projected/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-kube-api-access-v9xll\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:49.258274 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.258280    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511caeb6-5ade-4d40-bd25-8b8a6d6eea05-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:49.940105 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.940075    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l"
Nov 03 13:46:49.940305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.940118    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d324b0-kmw6l" event={"ID":"511caeb6-5ade-4d40-bd25-8b8a6d6eea05","Type":"ContainerDied","Data":"534ddd4bdba83d58502dc2a385bbc2693dcbf48d254d782ccc22ee4590fe820b"}
Nov 03 13:46:49.940305 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:49.940166    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534ddd4bdba83d58502dc2a385bbc2693dcbf48d254d782ccc22ee4590fe820b"
Nov 03 13:46:57.125402 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.125366    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"]
Nov 03 13:46:57.125915 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.125900    4546 cpu_manager.go:414] "containerMap: removing container" podUID="511caeb6-5ade-4d40-bd25-8b8a6d6eea05" containerName="revoke-keys"
Nov 03 13:46:57.125980 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.125917    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511caeb6-5ade-4d40-bd25-8b8a6d6eea05" containerName="revoke-keys"
Nov 03 13:46:57.125980 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.125977    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="511caeb6-5ade-4d40-bd25-8b8a6d6eea05" containerName="revoke-keys"
Nov 03 13:46:57.129210 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.129186    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.178526 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.178492    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:46:57.224801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.224761    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:46:57.224801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.224774    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:46:57.225115 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.224823    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:46:57.227169 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.227139    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"]
Nov 03 13:46:57.242429 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.242397    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.242603 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.242547    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gt2\" (UniqueName: \"kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.242678 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.242633    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.343221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.343187    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gt2\" (UniqueName: \"kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.343438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.343255    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.343438 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.343282    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.344007 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.343980    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.345936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.345917    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.407133 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.407057    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gt2\" (UniqueName: \"kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2\") pod \"maas-api-revoke-sub-b99d0fb8-wsgf4\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.443035 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.443003    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:46:57.808336 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.808311    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"]
Nov 03 13:46:57.810648 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:46:57.810617    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd149df_c2ec_4df3_bc61_2e22d1e97b70.slice/crio-addaaeb9f2c1901683872f8af11e02edabfdc377e4f7e3d21f53cf5d3d036aa6 WatchSource:0}: Error finding container addaaeb9f2c1901683872f8af11e02edabfdc377e4f7e3d21f53cf5d3d036aa6: Status 404 returned error can't find the container with id addaaeb9f2c1901683872f8af11e02edabfdc377e4f7e3d21f53cf5d3d036aa6
Nov 03 13:46:57.994422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:57.994386    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4" event={"ID":"7dd149df-c2ec-4df3-bc61-2e22d1e97b70","Type":"ContainerStarted","Data":"addaaeb9f2c1901683872f8af11e02edabfdc377e4f7e3d21f53cf5d3d036aa6"}
Nov 03 13:46:59.001282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:59.001247    4546 generic.go:356] "Generic (PLEG): container finished" podID="7dd149df-c2ec-4df3-bc61-2e22d1e97b70" containerID="7b7ff9a058f20c2db4aef71099a371eccdc0e6e349e828216c820747d166f988" exitCode=0
Nov 03 13:46:59.001680 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:46:59.001342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4" event={"ID":"7dd149df-c2ec-4df3-bc61-2e22d1e97b70","Type":"ContainerDied","Data":"7b7ff9a058f20c2db4aef71099a371eccdc0e6e349e828216c820747d166f988"}
Nov 03 13:47:00.142257 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.142231    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:47:00.272936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.272853    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2\" (UniqueName: \"kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2\") pod \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") "
Nov 03 13:47:00.273128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.272998    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca\") pod \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") "
Nov 03 13:47:00.273128 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.273052    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca\") pod \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\" (UID: \"7dd149df-c2ec-4df3-bc61-2e22d1e97b70\") "
Nov 03 13:47:00.273458 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.273429    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca" pod "7dd149df-c2ec-4df3-bc61-2e22d1e97b70" (UID: "7dd149df-c2ec-4df3-bc61-2e22d1e97b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:47:00.275192 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.275172    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2" pod "7dd149df-c2ec-4df3-bc61-2e22d1e97b70" (UID: "7dd149df-c2ec-4df3-bc61-2e22d1e97b70"). InnerVolumeSpecName "kube-api-access-f8gt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:47:00.275312 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.275277    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca" pod "7dd149df-c2ec-4df3-bc61-2e22d1e97b70" (UID: "7dd149df-c2ec-4df3-bc61-2e22d1e97b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:47:00.374170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.374102    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:00.374170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.374164    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gt2\" (UniqueName: \"kubernetes.io/projected/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-kube-api-access-f8gt2\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:00.374170 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:00.374178    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd149df-c2ec-4df3-bc61-2e22d1e97b70-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:01.012350 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:01.012315    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4"
Nov 03 13:47:01.012350 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:01.012342    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99d0fb8-wsgf4" event={"ID":"7dd149df-c2ec-4df3-bc61-2e22d1e97b70","Type":"ContainerDied","Data":"addaaeb9f2c1901683872f8af11e02edabfdc377e4f7e3d21f53cf5d3d036aa6"}
Nov 03 13:47:01.012575 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:01.012389    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="addaaeb9f2c1901683872f8af11e02edabfdc377e4f7e3d21f53cf5d3d036aa6"
Nov 03 13:47:03.641319 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:03.641278    4546 scope.go:122] "RemoveContainer" containerID="02e4a19246682244a406fffe4b4fe0dbd74dcf03a92f65ab204d26842fc2d73a"
Nov 03 13:47:07.651510 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:47:07.651423    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:47:07.920363 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.920287    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"]
Nov 03 13:47:07.920898 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.920879    4546 cpu_manager.go:414] "containerMap: removing container" podUID="7dd149df-c2ec-4df3-bc61-2e22d1e97b70" containerName="revoke-keys"
Nov 03 13:47:07.921000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.920902    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd149df-c2ec-4df3-bc61-2e22d1e97b70" containerName="revoke-keys"
Nov 03 13:47:07.921063 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.921051    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd149df-c2ec-4df3-bc61-2e22d1e97b70" containerName="revoke-keys"
Nov 03 13:47:07.926043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.926024    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:07.975077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.975041    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:47:07.975240 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.975122    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:47:07.975333 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.975272    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:47:07.975333 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.975304    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:47:07.977846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:07.977816    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"]
Nov 03 13:47:08.049158 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.049127    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.049353 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.049176    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.049353 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.049312    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv56\" (UniqueName: \"kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.150161 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.150118    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.150422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.150200    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv56\" (UniqueName: \"kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.150422 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.150247    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.150882 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.150859    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.152707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.152685    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.204403 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.204332    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv56\" (UniqueName: \"kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56\") pod \"maas-api-revoke-sub-75f0e3aa-7tkhm\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.243228 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.243196    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:08.620002 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:08.619978    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"]
Nov 03 13:47:08.621836 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:47:08.621804    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150b2442_47a8_43c2_864c_62fe1dfc298b.slice/crio-7bf369a60d0e5775c934094acba08407ccae0e5b4d8e2b6c7e0debadbeebafde WatchSource:0}: Error finding container 7bf369a60d0e5775c934094acba08407ccae0e5b4d8e2b6c7e0debadbeebafde: Status 404 returned error can't find the container with id 7bf369a60d0e5775c934094acba08407ccae0e5b4d8e2b6c7e0debadbeebafde
Nov 03 13:47:09.054652 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:09.054615    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm" event={"ID":"150b2442-47a8-43c2-864c-62fe1dfc298b","Type":"ContainerStarted","Data":"05954d23ef5348474143a76263d4b81393b308d40e8302707cfb744ab7b955ed"}
Nov 03 13:47:09.055154 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:09.054671    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm" event={"ID":"150b2442-47a8-43c2-864c-62fe1dfc298b","Type":"ContainerStarted","Data":"7bf369a60d0e5775c934094acba08407ccae0e5b4d8e2b6c7e0debadbeebafde"}
Nov 03 13:47:10.061325 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:10.061292    4546 generic.go:356] "Generic (PLEG): container finished" podID="150b2442-47a8-43c2-864c-62fe1dfc298b" containerID="05954d23ef5348474143a76263d4b81393b308d40e8302707cfb744ab7b955ed" exitCode=0
Nov 03 13:47:10.061879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:10.061371    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm" event={"ID":"150b2442-47a8-43c2-864c-62fe1dfc298b","Type":"ContainerDied","Data":"05954d23ef5348474143a76263d4b81393b308d40e8302707cfb744ab7b955ed"}
Nov 03 13:47:11.206278 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.206252    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:11.279829 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.279792    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56\" (UniqueName: \"kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56\") pod \"150b2442-47a8-43c2-864c-62fe1dfc298b\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") "
Nov 03 13:47:11.280030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.279838    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca\") pod \"150b2442-47a8-43c2-864c-62fe1dfc298b\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") "
Nov 03 13:47:11.280030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.279895    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca\") pod \"150b2442-47a8-43c2-864c-62fe1dfc298b\" (UID: \"150b2442-47a8-43c2-864c-62fe1dfc298b\") "
Nov 03 13:47:11.280334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.280307    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca" pod "150b2442-47a8-43c2-864c-62fe1dfc298b" (UID: "150b2442-47a8-43c2-864c-62fe1dfc298b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:47:11.282154 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.282130    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca" pod "150b2442-47a8-43c2-864c-62fe1dfc298b" (UID: "150b2442-47a8-43c2-864c-62fe1dfc298b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:47:11.282262 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.282147    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56" pod "150b2442-47a8-43c2-864c-62fe1dfc298b" (UID: "150b2442-47a8-43c2-864c-62fe1dfc298b"). InnerVolumeSpecName "kube-api-access-8sv56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:47:11.381239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.381153    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv56\" (UniqueName: \"kubernetes.io/projected/150b2442-47a8-43c2-864c-62fe1dfc298b-kube-api-access-8sv56\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:11.381239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.381186    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:11.381239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:11.381196    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150b2442-47a8-43c2-864c-62fe1dfc298b-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:12.073724 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:12.073682    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm" event={"ID":"150b2442-47a8-43c2-864c-62fe1dfc298b","Type":"ContainerDied","Data":"7bf369a60d0e5775c934094acba08407ccae0e5b4d8e2b6c7e0debadbeebafde"}
Nov 03 13:47:12.073952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:12.073776    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf369a60d0e5775c934094acba08407ccae0e5b4d8e2b6c7e0debadbeebafde"
Nov 03 13:47:12.073952 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:12.073703    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f0e3aa-7tkhm"
Nov 03 13:47:14.073775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:14.070087    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"]
Nov 03 13:47:14.091136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:14.091103    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glvcc"]
Nov 03 13:47:14.575720 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:14.575690    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5d3b74-c7c3-450e-846a-3a5989926d59" path="/var/lib/kubelet/pods/bd5d3b74-c7c3-450e-846a-3a5989926d59/volumes"
Nov 03 13:47:18.629428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.629389    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"]
Nov 03 13:47:18.629860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.629787    4546 cpu_manager.go:414] "containerMap: removing container" podUID="150b2442-47a8-43c2-864c-62fe1dfc298b" containerName="revoke-keys"
Nov 03 13:47:18.629860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.629800    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150b2442-47a8-43c2-864c-62fe1dfc298b" containerName="revoke-keys"
Nov 03 13:47:18.629951 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.629878    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="150b2442-47a8-43c2-864c-62fe1dfc298b" containerName="revoke-keys"
Nov 03 13:47:18.633495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.633474    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.684918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.684879    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:47:18.684918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.684895    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:47:18.731373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.731342    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:47:18.731373 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.731366    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:47:18.733875 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.733854    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"]
Nov 03 13:47:18.783179 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.783140    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.783357 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.783205    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftghw\" (UniqueName: \"kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.783400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.783379    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.884243 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.884137    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftghw\" (UniqueName: \"kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.884453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.884330    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.884453 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.884365    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.885165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.885147    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.887080 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.887063    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.937821 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.937786    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftghw\" (UniqueName: \"kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw\") pod \"maas-api-revoke-sub-ac5024ee-tpk5c\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:18.946618 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:18.946587    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:19.313279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:19.313254    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"]
Nov 03 13:47:19.314915 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:47:19.314886    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fda34a8_72d9_4b16_9551_4d6b24c31e80.slice/crio-4940e860b82a407894b6cb5df5d0af4979a9f9aca414e6fa94b22135b521d2d0 WatchSource:0}: Error finding container 4940e860b82a407894b6cb5df5d0af4979a9f9aca414e6fa94b22135b521d2d0: Status 404 returned error can't find the container with id 4940e860b82a407894b6cb5df5d0af4979a9f9aca414e6fa94b22135b521d2d0
Nov 03 13:47:20.123135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:20.123094    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c" event={"ID":"0fda34a8-72d9-4b16-9551-4d6b24c31e80","Type":"ContainerStarted","Data":"0bce07ecb89d3f6b77f6d5d066bd18ef94d99cf820788ab2632734a65b28b006"}
Nov 03 13:47:20.123610 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:20.123180    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c" event={"ID":"0fda34a8-72d9-4b16-9551-4d6b24c31e80","Type":"ContainerStarted","Data":"4940e860b82a407894b6cb5df5d0af4979a9f9aca414e6fa94b22135b521d2d0"}
Nov 03 13:47:20.231266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:20.231210    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c" podStartSLOduration=2.231194998 podStartE2EDuration="2.231194998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:20.229148791 +0000 UTC m=+6981.800732710" watchObservedRunningTime="2026-11-03 13:47:20.231194998 +0000 UTC m=+6981.802778909"
Nov 03 13:47:21.129913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:21.129878    4546 generic.go:356] "Generic (PLEG): container finished" podID="0fda34a8-72d9-4b16-9551-4d6b24c31e80" containerID="0bce07ecb89d3f6b77f6d5d066bd18ef94d99cf820788ab2632734a65b28b006" exitCode=0
Nov 03 13:47:21.130368 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:21.129968    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c" event={"ID":"0fda34a8-72d9-4b16-9551-4d6b24c31e80","Type":"ContainerDied","Data":"0bce07ecb89d3f6b77f6d5d066bd18ef94d99cf820788ab2632734a65b28b006"}
Nov 03 13:47:22.277522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.277495    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:22.419331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.419237    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca\") pod \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") "
Nov 03 13:47:22.419331 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.419311    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw\" (UniqueName: \"kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw\") pod \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") "
Nov 03 13:47:22.419594 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.419347    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca\") pod \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\" (UID: \"0fda34a8-72d9-4b16-9551-4d6b24c31e80\") "
Nov 03 13:47:22.419705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.419676    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca" pod "0fda34a8-72d9-4b16-9551-4d6b24c31e80" (UID: "0fda34a8-72d9-4b16-9551-4d6b24c31e80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:47:22.421694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.421666    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca" pod "0fda34a8-72d9-4b16-9551-4d6b24c31e80" (UID: "0fda34a8-72d9-4b16-9551-4d6b24c31e80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:47:22.421821 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.421774    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw" pod "0fda34a8-72d9-4b16-9551-4d6b24c31e80" (UID: "0fda34a8-72d9-4b16-9551-4d6b24c31e80"). InnerVolumeSpecName "kube-api-access-ftghw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:47:22.520166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.520135    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:22.520166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.520164    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda34a8-72d9-4b16-9551-4d6b24c31e80-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:22.520166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:22.520174    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftghw\" (UniqueName: \"kubernetes.io/projected/0fda34a8-72d9-4b16-9551-4d6b24c31e80-kube-api-access-ftghw\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:23.140873 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:23.140847    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c"
Nov 03 13:47:23.141055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:23.140875    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5024ee-tpk5c" event={"ID":"0fda34a8-72d9-4b16-9551-4d6b24c31e80","Type":"ContainerDied","Data":"4940e860b82a407894b6cb5df5d0af4979a9f9aca414e6fa94b22135b521d2d0"}
Nov 03 13:47:23.141055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:23.140919    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4940e860b82a407894b6cb5df5d0af4979a9f9aca414e6fa94b22135b521d2d0"
Nov 03 13:47:29.857670 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.857628    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"]
Nov 03 13:47:29.858221 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.858202    4546 cpu_manager.go:414] "containerMap: removing container" podUID="0fda34a8-72d9-4b16-9551-4d6b24c31e80" containerName="revoke-keys"
Nov 03 13:47:29.858367 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.858223    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fda34a8-72d9-4b16-9551-4d6b24c31e80" containerName="revoke-keys"
Nov 03 13:47:29.858434 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.858371    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fda34a8-72d9-4b16-9551-4d6b24c31e80" containerName="revoke-keys"
Nov 03 13:47:29.861736 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.861710    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.889416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.889381    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbc9\" (UniqueName: \"kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.889606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.889439    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.889606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.889554    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.958609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.958571    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:47:29.958609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.958597    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:47:29.958609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.958615    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:47:29.958609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.958609    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:47:29.961153 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.961123    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"]
Nov 03 13:47:29.990563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.990529    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbc9\" (UniqueName: \"kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.990775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.990573    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.990775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.990596    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.991308 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.991276    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:29.993152 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:29.993129    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:30.044269 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:30.044237    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbc9\" (UniqueName: \"kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9\") pod \"maas-api-revoke-sub-6635c992-249j4\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:30.176846 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:30.176739    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:30.547581 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:30.547552    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"]
Nov 03 13:47:30.549839 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:47:30.549802    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96c6128_f4a5_4628_9db0_053f318228a5.slice/crio-774d087119bc337a16a660c7d331e68ceb411cddce82c4f72c3b5c12adb5ed19 WatchSource:0}: Error finding container 774d087119bc337a16a660c7d331e68ceb411cddce82c4f72c3b5c12adb5ed19: Status 404 returned error can't find the container with id 774d087119bc337a16a660c7d331e68ceb411cddce82c4f72c3b5c12adb5ed19
Nov 03 13:47:31.190826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:31.190718    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4" event={"ID":"f96c6128-f4a5-4628-9db0-053f318228a5","Type":"ContainerStarted","Data":"bb7c42fdf770bf6a4bd65f35bf9f647e82dbb5960c8049ca5054f2e9bcb47f79"}
Nov 03 13:47:31.190826 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:31.190803    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4" event={"ID":"f96c6128-f4a5-4628-9db0-053f318228a5","Type":"ContainerStarted","Data":"774d087119bc337a16a660c7d331e68ceb411cddce82c4f72c3b5c12adb5ed19"}
Nov 03 13:47:31.297595 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:31.297538    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4" podStartSLOduration=2.297522347 podStartE2EDuration="2.297522347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:31.296860874 +0000 UTC m=+6992.868444803" watchObservedRunningTime="2026-11-03 13:47:31.297522347 +0000 UTC m=+6992.869106258"
Nov 03 13:47:32.198001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:32.197960    4546 generic.go:356] "Generic (PLEG): container finished" podID="f96c6128-f4a5-4628-9db0-053f318228a5" containerID="bb7c42fdf770bf6a4bd65f35bf9f647e82dbb5960c8049ca5054f2e9bcb47f79" exitCode=0
Nov 03 13:47:32.198425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:32.198054    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4" event={"ID":"f96c6128-f4a5-4628-9db0-053f318228a5","Type":"ContainerDied","Data":"bb7c42fdf770bf6a4bd65f35bf9f647e82dbb5960c8049ca5054f2e9bcb47f79"}
Nov 03 13:47:33.343916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.343889    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:33.424055 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.424012    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca\") pod \"f96c6128-f4a5-4628-9db0-053f318228a5\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") "
Nov 03 13:47:33.424280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.424188    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9\" (UniqueName: \"kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9\") pod \"f96c6128-f4a5-4628-9db0-053f318228a5\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") "
Nov 03 13:47:33.424280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.424230    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca\") pod \"f96c6128-f4a5-4628-9db0-053f318228a5\" (UID: \"f96c6128-f4a5-4628-9db0-053f318228a5\") "
Nov 03 13:47:33.424651 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.424621    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca" pod "f96c6128-f4a5-4628-9db0-053f318228a5" (UID: "f96c6128-f4a5-4628-9db0-053f318228a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:47:33.426834 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.426806    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca" pod "f96c6128-f4a5-4628-9db0-053f318228a5" (UID: "f96c6128-f4a5-4628-9db0-053f318228a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:47:33.426956 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.426906    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9" pod "f96c6128-f4a5-4628-9db0-053f318228a5" (UID: "f96c6128-f4a5-4628-9db0-053f318228a5"). InnerVolumeSpecName "kube-api-access-clbc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:47:33.525974 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.525932    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:33.525974 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.525974    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbc9\" (UniqueName: \"kubernetes.io/projected/f96c6128-f4a5-4628-9db0-053f318228a5-kube-api-access-clbc9\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:33.526252 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:33.525993    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c6128-f4a5-4628-9db0-053f318228a5-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:34.211266 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:34.211220    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4" event={"ID":"f96c6128-f4a5-4628-9db0-053f318228a5","Type":"ContainerDied","Data":"774d087119bc337a16a660c7d331e68ceb411cddce82c4f72c3b5c12adb5ed19"}
Nov 03 13:47:34.211539 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:34.211278    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="774d087119bc337a16a660c7d331e68ceb411cddce82c4f72c3b5c12adb5ed19"
Nov 03 13:47:34.211539 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:34.211282    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6635c992-249j4"
Nov 03 13:47:36.047869 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:36.047815    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"]
Nov 03 13:47:36.093633 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:36.093595    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nk7gh"]
Nov 03 13:47:36.571523 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:36.571493    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d700e5dd-004d-4561-b77c-e12fc14854d6" path="/var/lib/kubelet/pods/d700e5dd-004d-4561-b77c-e12fc14854d6/volumes"
Nov 03 13:47:37.649964 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:47:37.649928    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:47:40.613343 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.613298    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"]
Nov 03 13:47:40.613913 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.613893    4546 cpu_manager.go:414] "containerMap: removing container" podUID="f96c6128-f4a5-4628-9db0-053f318228a5" containerName="revoke-keys"
Nov 03 13:47:40.614008 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.613915    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96c6128-f4a5-4628-9db0-053f318228a5" containerName="revoke-keys"
Nov 03 13:47:40.614070 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.614028    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96c6128-f4a5-4628-9db0-053f318228a5" containerName="revoke-keys"
Nov 03 13:47:40.617496 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.617472    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.666334 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.666297    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:47:40.712761 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.712709    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:47:40.712940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.712789    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:47:40.712940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.712713    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:47:40.715252 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.715233    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"]
Nov 03 13:47:40.721390 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.721363    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.721541 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.721413    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.721541 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.721530    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhkv\" (UniqueName: \"kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.822214 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.822171    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.822413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.822227    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhkv\" (UniqueName: \"kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.822413 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.822301    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.822887 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.822863    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.824791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.824767    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.876372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.876294    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhkv\" (UniqueName: \"kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv\") pod \"maas-api-revoke-sub-cbd40680-7wqxs\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:40.929905 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:40.929867    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:41.297277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:41.297247    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"]
Nov 03 13:47:41.299177 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:47:41.299146    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc339f1d5_217b_4c3e_a4bc_91ea7d91df63.slice/crio-c41a12e60e1c40ab795de5fa35b189cc6a461f4d6c533ede5f5e7430d26dcad4 WatchSource:0}: Error finding container c41a12e60e1c40ab795de5fa35b189cc6a461f4d6c533ede5f5e7430d26dcad4: Status 404 returned error can't find the container with id c41a12e60e1c40ab795de5fa35b189cc6a461f4d6c533ede5f5e7430d26dcad4
Nov 03 13:47:42.279876 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:42.279840    4546 generic.go:356] "Generic (PLEG): container finished" podID="c339f1d5-217b-4c3e-a4bc-91ea7d91df63" containerID="ef54962d2805185f3a6348f733c47bc9f08849485e69d682c4bc74630ab5cc9c" exitCode=0
Nov 03 13:47:42.280314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:42.279919    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs" event={"ID":"c339f1d5-217b-4c3e-a4bc-91ea7d91df63","Type":"ContainerDied","Data":"ef54962d2805185f3a6348f733c47bc9f08849485e69d682c4bc74630ab5cc9c"}
Nov 03 13:47:42.280314 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:42.279970    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs" event={"ID":"c339f1d5-217b-4c3e-a4bc-91ea7d91df63","Type":"ContainerStarted","Data":"c41a12e60e1c40ab795de5fa35b189cc6a461f4d6c533ede5f5e7430d26dcad4"}
Nov 03 13:47:43.413870 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.413845    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:43.548697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.548606    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca\") pod \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") "
Nov 03 13:47:43.548697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.548667    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv\" (UniqueName: \"kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv\") pod \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") "
Nov 03 13:47:43.548697 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.548689    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca\") pod \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\" (UID: \"c339f1d5-217b-4c3e-a4bc-91ea7d91df63\") "
Nov 03 13:47:43.549121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.549095    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca" pod "c339f1d5-217b-4c3e-a4bc-91ea7d91df63" (UID: "c339f1d5-217b-4c3e-a4bc-91ea7d91df63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:47:43.551010 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.550983    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca" pod "c339f1d5-217b-4c3e-a4bc-91ea7d91df63" (UID: "c339f1d5-217b-4c3e-a4bc-91ea7d91df63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:47:43.551099 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.551040    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv" pod "c339f1d5-217b-4c3e-a4bc-91ea7d91df63" (UID: "c339f1d5-217b-4c3e-a4bc-91ea7d91df63"). InnerVolumeSpecName "kube-api-access-zzhkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:47:43.650233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.650196    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:43.650233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.650226    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhkv\" (UniqueName: \"kubernetes.io/projected/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-kube-api-access-zzhkv\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:43.650233 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:43.650238    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c339f1d5-217b-4c3e-a4bc-91ea7d91df63-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:44.291282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:44.291250    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs"
Nov 03 13:47:44.291476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:44.291283    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd40680-7wqxs" event={"ID":"c339f1d5-217b-4c3e-a4bc-91ea7d91df63","Type":"ContainerDied","Data":"c41a12e60e1c40ab795de5fa35b189cc6a461f4d6c533ede5f5e7430d26dcad4"}
Nov 03 13:47:44.291476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:44.291344    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41a12e60e1c40ab795de5fa35b189cc6a461f4d6c533ede5f5e7430d26dcad4"
Nov 03 13:47:51.522396 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.522360    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"]
Nov 03 13:47:51.522844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.522773    4546 cpu_manager.go:414] "containerMap: removing container" podUID="c339f1d5-217b-4c3e-a4bc-91ea7d91df63" containerName="revoke-keys"
Nov 03 13:47:51.522844 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.522786    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c339f1d5-217b-4c3e-a4bc-91ea7d91df63" containerName="revoke-keys"
Nov 03 13:47:51.522946 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.522854    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="c339f1d5-217b-4c3e-a4bc-91ea7d91df63" containerName="revoke-keys"
Nov 03 13:47:51.526359 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.526334    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.577464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.577427    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:47:51.577464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.577437    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:47:51.623527 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.623484    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:47:51.623726 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.623560    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:47:51.626309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.626287    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"]
Nov 03 13:47:51.627597 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.627575    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xw6r\" (UniqueName: \"kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.627771 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.627733    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.627863 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.627781    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.728988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.728941    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xw6r\" (UniqueName: \"kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.729160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.729043    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.729160 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.729072    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.729900 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.729868    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.731700 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.731683    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.782988 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.782903    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xw6r\" (UniqueName: \"kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r\") pod \"maas-api-revoke-sub-08c85cf3-85rx9\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:51.841864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:51.841829    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:52.211674 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:52.211647    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"]
Nov 03 13:47:52.213557 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:47:52.213524    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40232fcc_7390_418c_af98_625841124ae6.slice/crio-9fcac9cf627b513fc11e4eff221cc6af8a4f8e1c250cca22b20c8ca7f91d4459 WatchSource:0}: Error finding container 9fcac9cf627b513fc11e4eff221cc6af8a4f8e1c250cca22b20c8ca7f91d4459: Status 404 returned error can't find the container with id 9fcac9cf627b513fc11e4eff221cc6af8a4f8e1c250cca22b20c8ca7f91d4459
Nov 03 13:47:52.330664 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:52.330625    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9" event={"ID":"40232fcc-7390-418c-af98-625841124ae6","Type":"ContainerStarted","Data":"9fcac9cf627b513fc11e4eff221cc6af8a4f8e1c250cca22b20c8ca7f91d4459"}
Nov 03 13:47:53.337143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:53.337052    4546 generic.go:356] "Generic (PLEG): container finished" podID="40232fcc-7390-418c-af98-625841124ae6" containerID="9ff6db119f88be8541b121697a01d5356f385ecbe423dd810410adf8d2b943f3" exitCode=0
Nov 03 13:47:53.337143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:53.337130    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9" event={"ID":"40232fcc-7390-418c-af98-625841124ae6","Type":"ContainerDied","Data":"9ff6db119f88be8541b121697a01d5356f385ecbe423dd810410adf8d2b943f3"}
Nov 03 13:47:54.515888 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.515851    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:47:54.656878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.656768    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca\") pod \"40232fcc-7390-418c-af98-625841124ae6\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") "
Nov 03 13:47:54.656878 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.656834    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r\" (UniqueName: \"kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r\") pod \"40232fcc-7390-418c-af98-625841124ae6\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") "
Nov 03 13:47:54.657159 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.656984    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca\") pod \"40232fcc-7390-418c-af98-625841124ae6\" (UID: \"40232fcc-7390-418c-af98-625841124ae6\") "
Nov 03 13:47:54.658574 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.658539    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca" pod "40232fcc-7390-418c-af98-625841124ae6" (UID: "40232fcc-7390-418c-af98-625841124ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:47:54.659658 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.659632    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca" pod "40232fcc-7390-418c-af98-625841124ae6" (UID: "40232fcc-7390-418c-af98-625841124ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:47:54.661780 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.661708    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r" pod "40232fcc-7390-418c-af98-625841124ae6" (UID: "40232fcc-7390-418c-af98-625841124ae6"). InnerVolumeSpecName "kube-api-access-4xw6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:47:54.759509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.759403    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40232fcc-7390-418c-af98-625841124ae6-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:54.759509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.759452    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40232fcc-7390-418c-af98-625841124ae6-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:54.759509 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:54.759469    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xw6r\" (UniqueName: \"kubernetes.io/projected/40232fcc-7390-418c-af98-625841124ae6-kube-api-access-4xw6r\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:47:55.374774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:55.373882    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9" event={"ID":"40232fcc-7390-418c-af98-625841124ae6","Type":"ContainerDied","Data":"9fcac9cf627b513fc11e4eff221cc6af8a4f8e1c250cca22b20c8ca7f91d4459"}
Nov 03 13:47:55.374774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:55.373954    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fcac9cf627b513fc11e4eff221cc6af8a4f8e1c250cca22b20c8ca7f91d4459"
Nov 03 13:47:55.374774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:47:55.374032    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08c85cf3-85rx9"
Nov 03 13:48:02.318120 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.318077    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"]
Nov 03 13:48:02.318548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.318456    4546 cpu_manager.go:414] "containerMap: removing container" podUID="40232fcc-7390-418c-af98-625841124ae6" containerName="revoke-keys"
Nov 03 13:48:02.318548 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.318466    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40232fcc-7390-418c-af98-625841124ae6" containerName="revoke-keys"
Nov 03 13:48:02.318631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.318563    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="40232fcc-7390-418c-af98-625841124ae6" containerName="revoke-keys"
Nov 03 13:48:02.321893 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.321871    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.419141 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.419098    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:48:02.419372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.419175    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:48:02.419372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.419204    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:48:02.419372 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.419225    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:48:02.422135 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.422108    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"]
Nov 03 13:48:02.450939 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.450899    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.451101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.450955    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.451177 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.451104    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkbh\" (UniqueName: \"kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.552079 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.552038    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.552280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.552097    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.552280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.552165    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkbh\" (UniqueName: \"kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.552730 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.552695    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.554798 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.554776    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.606231 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.606159    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkbh\" (UniqueName: \"kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh\") pod \"maas-api-revoke-sub-2dde88c6-nxmsg\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:02.637703 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:02.637672    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:03.003460 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:03.003433    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"]
Nov 03 13:48:03.008712 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:48:03.008676    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc235bb_497e_4092_a147_33d14b56da64.slice/crio-1448f0b4315dfa9cf9d9ac3008f56e8da029594ff916c80451c98b503c78aeb1 WatchSource:0}: Error finding container 1448f0b4315dfa9cf9d9ac3008f56e8da029594ff916c80451c98b503c78aeb1: Status 404 returned error can't find the container with id 1448f0b4315dfa9cf9d9ac3008f56e8da029594ff916c80451c98b503c78aeb1
Nov 03 13:48:03.421659 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:03.421617    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg" event={"ID":"9fc235bb-497e-4092-a147-33d14b56da64","Type":"ContainerStarted","Data":"89f5f30d6fd0970c953f895379331b55cf3cfdc91c1f14fd234fdc7aa8050ac4"}
Nov 03 13:48:03.422101 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:03.421684    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg" event={"ID":"9fc235bb-497e-4092-a147-33d14b56da64","Type":"ContainerStarted","Data":"1448f0b4315dfa9cf9d9ac3008f56e8da029594ff916c80451c98b503c78aeb1"}
Nov 03 13:48:03.574797 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:03.574625    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg" podStartSLOduration=1.574611011 podStartE2EDuration="1.574611011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:03.527951384 +0000 UTC m=+7025.099535316" watchObservedRunningTime="2026-11-03 13:48:03.574611011 +0000 UTC m=+7025.146194966"
Nov 03 13:48:04.428256 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:04.428213    4546 generic.go:356] "Generic (PLEG): container finished" podID="9fc235bb-497e-4092-a147-33d14b56da64" containerID="89f5f30d6fd0970c953f895379331b55cf3cfdc91c1f14fd234fdc7aa8050ac4" exitCode=0
Nov 03 13:48:04.428699 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:04.428309    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg" event={"ID":"9fc235bb-497e-4092-a147-33d14b56da64","Type":"ContainerDied","Data":"89f5f30d6fd0970c953f895379331b55cf3cfdc91c1f14fd234fdc7aa8050ac4"}
Nov 03 13:48:05.599534 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.599499    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:05.686472 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.686342    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca\") pod \"9fc235bb-497e-4092-a147-33d14b56da64\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") "
Nov 03 13:48:05.687965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.686849    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh\" (UniqueName: \"kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh\") pod \"9fc235bb-497e-4092-a147-33d14b56da64\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") "
Nov 03 13:48:05.687965 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.686941    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca\") pod \"9fc235bb-497e-4092-a147-33d14b56da64\" (UID: \"9fc235bb-497e-4092-a147-33d14b56da64\") "
Nov 03 13:48:05.688440 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.688413    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca" pod "9fc235bb-497e-4092-a147-33d14b56da64" (UID: "9fc235bb-497e-4092-a147-33d14b56da64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:48:05.690984 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.690944    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh" pod "9fc235bb-497e-4092-a147-33d14b56da64" (UID: "9fc235bb-497e-4092-a147-33d14b56da64"). InnerVolumeSpecName "kube-api-access-gqkbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:48:05.691235 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.691189    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca" pod "9fc235bb-497e-4092-a147-33d14b56da64" (UID: "9fc235bb-497e-4092-a147-33d14b56da64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:48:05.788670 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.788625    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:05.789024 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.788998    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqkbh\" (UniqueName: \"kubernetes.io/projected/9fc235bb-497e-4092-a147-33d14b56da64-kube-api-access-gqkbh\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:05.789164 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:05.789150    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc235bb-497e-4092-a147-33d14b56da64-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:06.453778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:06.453038    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg"
Nov 03 13:48:06.469776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:06.466903    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dde88c6-nxmsg" event={"ID":"9fc235bb-497e-4092-a147-33d14b56da64","Type":"ContainerDied","Data":"1448f0b4315dfa9cf9d9ac3008f56e8da029594ff916c80451c98b503c78aeb1"}
Nov 03 13:48:06.469776 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:06.466978    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1448f0b4315dfa9cf9d9ac3008f56e8da029594ff916c80451c98b503c78aeb1"
Nov 03 13:48:07.651425 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:48:07.651389    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:48:13.106899 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.106861    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"]
Nov 03 13:48:13.107407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.107281    4546 cpu_manager.go:414] "containerMap: removing container" podUID="9fc235bb-497e-4092-a147-33d14b56da64" containerName="revoke-keys"
Nov 03 13:48:13.107407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.107292    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc235bb-497e-4092-a147-33d14b56da64" containerName="revoke-keys"
Nov 03 13:48:13.107407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.107370    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc235bb-497e-4092-a147-33d14b56da64" containerName="revoke-keys"
Nov 03 13:48:13.110849 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.110828    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.160061 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.160021    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.160240 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.160092    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.160240 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.160213    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpsq\" (UniqueName: \"kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.164861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.164831    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"]
Nov 03 13:48:13.208495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.208456    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:48:13.208495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.208466    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:48:13.208729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.208506    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:48:13.208729 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.208522    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:48:13.261428 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.261389    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.261648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.261443    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.261648 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.261509    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpsq\" (UniqueName: \"kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.262225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.262196    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.264032 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.264014    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.319095 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.319061    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpsq\" (UniqueName: \"kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq\") pod \"maas-api-revoke-sub-7a7963b9-ss5rq\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.424077 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.423977    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:13.790859 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:13.790834    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"]
Nov 03 13:48:13.792300 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:48:13.792271    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4b859f_8b05_4cf2_b55f_a3c12987bbda.slice/crio-8bd203609c8a041f1005214d434ccc40ce4a4635bcb962af0512259a0896d994 WatchSource:0}: Error finding container 8bd203609c8a041f1005214d434ccc40ce4a4635bcb962af0512259a0896d994: Status 404 returned error can't find the container with id 8bd203609c8a041f1005214d434ccc40ce4a4635bcb962af0512259a0896d994
Nov 03 13:48:14.054860 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:14.054777    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"]
Nov 03 13:48:14.057058 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:14.057035    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ceb754-krnhp"]
Nov 03 13:48:14.510365 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:14.510328    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq" event={"ID":"5c4b859f-8b05-4cf2-b55f-a3c12987bbda","Type":"ContainerStarted","Data":"5244a3fb46648e29d98123edad00b845fdb42b034431e87a969343f1d1c901e6"}
Nov 03 13:48:14.510861 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:14.510384    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq" event={"ID":"5c4b859f-8b05-4cf2-b55f-a3c12987bbda","Type":"ContainerStarted","Data":"8bd203609c8a041f1005214d434ccc40ce4a4635bcb962af0512259a0896d994"}
Nov 03 13:48:14.571839 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:14.571804    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4756ee1e-98ec-4841-a9e1-dd6b04383dab" path="/var/lib/kubelet/pods/4756ee1e-98ec-4841-a9e1-dd6b04383dab/volumes"
Nov 03 13:48:14.618847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:14.618794    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq" podStartSLOduration=1.6187787999999999 podStartE2EDuration="1.6187788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:14.617544753 +0000 UTC m=+7036.189128690" watchObservedRunningTime="2026-11-03 13:48:14.6187788 +0000 UTC m=+7036.190362709"
Nov 03 13:48:15.518504 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:15.518465    4546 generic.go:356] "Generic (PLEG): container finished" podID="5c4b859f-8b05-4cf2-b55f-a3c12987bbda" containerID="5244a3fb46648e29d98123edad00b845fdb42b034431e87a969343f1d1c901e6" exitCode=0
Nov 03 13:48:15.519116 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:15.518586    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq" event={"ID":"5c4b859f-8b05-4cf2-b55f-a3c12987bbda","Type":"ContainerDied","Data":"5244a3fb46648e29d98123edad00b845fdb42b034431e87a969343f1d1c901e6"}
Nov 03 13:48:17.006592 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.006569    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:17.102282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.102248    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca\") pod \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") "
Nov 03 13:48:17.102476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.102357    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq\" (UniqueName: \"kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq\") pod \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") "
Nov 03 13:48:17.102476 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.102438    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca\") pod \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\" (UID: \"5c4b859f-8b05-4cf2-b55f-a3c12987bbda\") "
Nov 03 13:48:17.102630 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.102608    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca" pod "5c4b859f-8b05-4cf2-b55f-a3c12987bbda" (UID: "5c4b859f-8b05-4cf2-b55f-a3c12987bbda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:48:17.102808 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.102784    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:17.104711 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.104686    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca" pod "5c4b859f-8b05-4cf2-b55f-a3c12987bbda" (UID: "5c4b859f-8b05-4cf2-b55f-a3c12987bbda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:48:17.104830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.104786    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq" pod "5c4b859f-8b05-4cf2-b55f-a3c12987bbda" (UID: "5c4b859f-8b05-4cf2-b55f-a3c12987bbda"). InnerVolumeSpecName "kube-api-access-7rpsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:48:17.204187 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.204135    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rpsq\" (UniqueName: \"kubernetes.io/projected/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-kube-api-access-7rpsq\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:17.204187 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.204164    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b859f-8b05-4cf2-b55f-a3c12987bbda-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:17.590783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.587866    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq" event={"ID":"5c4b859f-8b05-4cf2-b55f-a3c12987bbda","Type":"ContainerDied","Data":"8bd203609c8a041f1005214d434ccc40ce4a4635bcb962af0512259a0896d994"}
Nov 03 13:48:17.590783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.587945    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd203609c8a041f1005214d434ccc40ce4a4635bcb962af0512259a0896d994"
Nov 03 13:48:17.590783 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.588035    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7963b9-ss5rq"
Nov 03 13:48:17.689971 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.689908    4546 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zg2zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:48:17.690360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:17.690304    4546 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zg2zd" podUID="45e0aa94-18d0-4583-8748-75f6100b6272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:48:23.901111 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.901075    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"]
Nov 03 13:48:23.901540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.901492    4546 cpu_manager.go:414] "containerMap: removing container" podUID="5c4b859f-8b05-4cf2-b55f-a3c12987bbda" containerName="revoke-keys"
Nov 03 13:48:23.901540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.901502    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4b859f-8b05-4cf2-b55f-a3c12987bbda" containerName="revoke-keys"
Nov 03 13:48:23.901616 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.901572    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4b859f-8b05-4cf2-b55f-a3c12987bbda" containerName="revoke-keys"
Nov 03 13:48:23.905415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.905386    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:23.959155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.959114    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"]
Nov 03 13:48:23.989570 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.989538    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:23.989774 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.989591    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqst\" (UniqueName: \"kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:23.989830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:23.989741    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.002122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.002093    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:48:24.002299 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.002280    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:48:24.002360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.002298    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:48:24.002360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.002330    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:48:24.091119 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.091083    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqst\" (UniqueName: \"kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.091327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.091149    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.091327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.091257    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.091830 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.091796    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.093801 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.093777    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.145041 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.145005    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqst\" (UniqueName: \"kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst\") pod \"maas-api-revoke-sub-7f01da68-5vwqd\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.221969 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.221943    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:24.587800 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.587778    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"]
Nov 03 13:48:24.589233 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:48:24.589204    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c1d268_5e97_432f_b63f_ad41d3d2f4d6.slice/crio-0babb20ec00690b2444bc9eb65a38b69f4f1211c85f087ef34f9c4bd05e85de4 WatchSource:0}: Error finding container 0babb20ec00690b2444bc9eb65a38b69f4f1211c85f087ef34f9c4bd05e85de4: Status 404 returned error can't find the container with id 0babb20ec00690b2444bc9eb65a38b69f4f1211c85f087ef34f9c4bd05e85de4
Nov 03 13:48:24.624172 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:24.624138    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd" event={"ID":"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6","Type":"ContainerStarted","Data":"0babb20ec00690b2444bc9eb65a38b69f4f1211c85f087ef34f9c4bd05e85de4"}
Nov 03 13:48:25.044027 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:25.043987    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"]
Nov 03 13:48:25.068358 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:25.068324    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43a23ab-58vx7"]
Nov 03 13:48:25.631637 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:25.631607    4546 generic.go:356] "Generic (PLEG): container finished" podID="b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" containerID="ab30ec838b7d60f910ff0f49ff7e7479eb54bc844dd05c7cdf5fa2c39f7101a2" exitCode=0
Nov 03 13:48:25.631872 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:25.631695    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd" event={"ID":"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6","Type":"ContainerDied","Data":"ab30ec838b7d60f910ff0f49ff7e7479eb54bc844dd05c7cdf5fa2c39f7101a2"}
Nov 03 13:48:26.574143 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.574100    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b9e71e-cf93-428e-93c5-12ad553b3e91" path="/var/lib/kubelet/pods/90b9e71e-cf93-428e-93c5-12ad553b3e91/volumes"
Nov 03 13:48:26.808464 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.808435    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:26.921085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.920990    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca\") pod \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") "
Nov 03 13:48:26.921085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.921056    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst\" (UniqueName: \"kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst\") pod \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") "
Nov 03 13:48:26.921347 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.921162    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca\") pod \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\" (UID: \"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6\") "
Nov 03 13:48:26.921918 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.921888    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca" pod "b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" (UID: "b9c1d268-5e97-432f-b63f-ad41d3d2f4d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:48:26.934425 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.934372    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst" pod "b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" (UID: "b9c1d268-5e97-432f-b63f-ad41d3d2f4d6"). InnerVolumeSpecName "kube-api-access-zlqst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:48:26.934638 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:26.934607    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca" pod "b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" (UID: "b9c1d268-5e97-432f-b63f-ad41d3d2f4d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:48:27.022414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:27.022370    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:27.022414 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:27.022418    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqst\" (UniqueName: \"kubernetes.io/projected/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-kube-api-access-zlqst\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:27.022685 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:27.022436    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1d268-5e97-432f-b63f-ad41d3d2f4d6-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:27.661775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:27.661625    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd" event={"ID":"b9c1d268-5e97-432f-b63f-ad41d3d2f4d6","Type":"ContainerDied","Data":"0babb20ec00690b2444bc9eb65a38b69f4f1211c85f087ef34f9c4bd05e85de4"}
Nov 03 13:48:27.661775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:27.661699    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0babb20ec00690b2444bc9eb65a38b69f4f1211c85f087ef34f9c4bd05e85de4"
Nov 03 13:48:27.668775 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:27.665978    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01da68-5vwqd"
Nov 03 13:48:34.717617 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.717581    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"]
Nov 03 13:48:34.718121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.717990    4546 cpu_manager.go:414] "containerMap: removing container" podUID="b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" containerName="revoke-keys"
Nov 03 13:48:34.718121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.718001    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" containerName="revoke-keys"
Nov 03 13:48:34.718121 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.718065    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c1d268-5e97-432f-b63f-ad41d3d2f4d6" containerName="revoke-keys"
Nov 03 13:48:34.721325 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.721306    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.738345 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.738314    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.738500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.738434    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.738500 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.738470    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm65t\" (UniqueName: \"kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.770962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.770927    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:48:34.770962 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.770927    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:48:34.817155 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.817115    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:48:34.817371 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.817222    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:48:34.819864 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.819839    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"]
Nov 03 13:48:34.839495 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.839472    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.839585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.839500    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm65t\" (UniqueName: \"kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.839585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.839558    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.840125 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.840105    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.842045 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.842022    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:34.894215 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:34.894177    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm65t\" (UniqueName: \"kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t\") pod \"maas-api-revoke-sub-d61ce2c5-ptshp\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:35.037644 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:35.037556    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:35.406992 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:35.406968    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"]
Nov 03 13:48:35.408928 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:48:35.408898    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fabee64_f57b_40c7_acc7_1dfdee1fcb47.slice/crio-e7a20023b9148f96c5c85d6e2faaf8407b91c746720b344eb500f6877fd9992b WatchSource:0}: Error finding container e7a20023b9148f96c5c85d6e2faaf8407b91c746720b344eb500f6877fd9992b: Status 404 returned error can't find the container with id e7a20023b9148f96c5c85d6e2faaf8407b91c746720b344eb500f6877fd9992b
Nov 03 13:48:35.733881 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:35.733836    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp" event={"ID":"1fabee64-f57b-40c7-acc7-1dfdee1fcb47","Type":"ContainerStarted","Data":"e7a20023b9148f96c5c85d6e2faaf8407b91c746720b344eb500f6877fd9992b"}
Nov 03 13:48:36.740960 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:36.740919    4546 generic.go:356] "Generic (PLEG): container finished" podID="1fabee64-f57b-40c7-acc7-1dfdee1fcb47" containerID="fa066527821261a44cb4e362b891b230fe900a171a8aa26f133530e2bd2c3307" exitCode=0
Nov 03 13:48:36.741407 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:36.741007    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp" event={"ID":"1fabee64-f57b-40c7-acc7-1dfdee1fcb47","Type":"ContainerDied","Data":"fa066527821261a44cb4e362b891b230fe900a171a8aa26f133530e2bd2c3307"}
Nov 03 13:48:37.699094 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:48:37.699008    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:48:38.157400 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.157364    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:38.192903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.192862    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t\" (UniqueName: \"kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t\") pod \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") "
Nov 03 13:48:38.193095 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.192968    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca\") pod \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") "
Nov 03 13:48:38.193177 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.193151    4546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca\") pod \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\" (UID: \"1fabee64-f57b-40c7-acc7-1dfdee1fcb47\") "
Nov 03 13:48:38.193564 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.193537    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca" pod "1fabee64-f57b-40c7-acc7-1dfdee1fcb47" (UID: "1fabee64-f57b-40c7-acc7-1dfdee1fcb47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:48:38.195596 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.195572    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca" pod "1fabee64-f57b-40c7-acc7-1dfdee1fcb47" (UID: "1fabee64-f57b-40c7-acc7-1dfdee1fcb47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:48:38.196136 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.196111    4546 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t" pod "1fabee64-f57b-40c7-acc7-1dfdee1fcb47" (UID: "1fabee64-f57b-40c7-acc7-1dfdee1fcb47"). InnerVolumeSpecName "kube-api-access-sm65t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:48:38.294307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.294270    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-openshift-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:38.294307 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.294304    4546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm65t\" (UniqueName: \"kubernetes.io/projected/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-kube-api-access-sm65t\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:38.294561 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.294318    4546 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fabee64-f57b-40c7-acc7-1dfdee1fcb47-maas-api-xks-ca\") on node \"ip-10-0-136-245.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:48:38.777279 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.777223    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp" event={"ID":"1fabee64-f57b-40c7-acc7-1dfdee1fcb47","Type":"ContainerDied","Data":"e7a20023b9148f96c5c85d6e2faaf8407b91c746720b344eb500f6877fd9992b"}
Nov 03 13:48:38.777602 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.777566    4546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a20023b9148f96c5c85d6e2faaf8407b91c746720b344eb500f6877fd9992b"
Nov 03 13:48:38.777968 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:38.777941    4546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61ce2c5-ptshp"
Nov 03 13:48:59.048557 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:59.048516    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"]
Nov 03 13:48:59.094144 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:48:59.094107    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6414fe74-rncmf"]
Nov 03 13:49:00.572122 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:00.572085    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03198087-5a86-405f-936f-29c9787410cb" path="/var/lib/kubelet/pods/03198087-5a86-405f-936f-29c9787410cb/volumes"
Nov 03 13:49:03.673467 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:03.673434    4546 scope.go:122] "RemoveContainer" containerID="46866e009cfa81a8e8ae502a625f06a877a517a0128370064d45199fc8572be8"
Nov 03 13:49:03.682327 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:03.682304    4546 scope.go:122] "RemoveContainer" containerID="1814a8cbf27cd1dc2a6da54b9ed99548d6e7672797d7cdf31705e3a6f457d376"
Nov 03 13:49:07.651153 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:49:07.651115    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:49:14.599879 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:14.599843    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cd79d8544-f6nxm_abc3e369-07e6-4332-a68a-58c77461df97/manager/0.log"
Nov 03 13:49:16.154990 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:16.154960    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c58c9bfd-64vqd_e794e88c-e1c1-4534-9bb5-bbc0a8c40773/manager/0.log"
Nov 03 13:49:18.691484 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:18.691452    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-08c85cf3-85rx9_40232fcc-7390-418c-af98-625841124ae6/revoke-keys/0.log"
Nov 03 13:49:18.907011 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:18.906977    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2dde88c6-nxmsg_9fc235bb-497e-4092-a147-33d14b56da64/revoke-keys/0.log"
Nov 03 13:49:19.128791 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:19.128744    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-48061e9f-xs8pq_01463f96-983a-4f86-b409-c91e1e09e51e/revoke-keys/0.log"
Nov 03 13:49:19.352273 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:19.352242    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-55f36919-t9znd_ecb8ff4e-b8ff-4b13-8720-da6cc4a8400e/revoke-keys/0.log"
Nov 03 13:49:19.569165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:19.569135    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-58d324b0-kmw6l_511caeb6-5ade-4d40-bd25-8b8a6d6eea05/revoke-keys/0.log"
Nov 03 13:49:19.797354 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:19.797326    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6635c992-249j4_f96c6128-f4a5-4628-9db0-053f318228a5/revoke-keys/0.log"
Nov 03 13:49:20.023178 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:20.023146    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6ec59d4a-psd4r_819e7513-46bc-4f61-b929-85e7027d777c/revoke-keys/0.log"
Nov 03 13:49:20.236955 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:20.236916    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-75f0e3aa-7tkhm_150b2442-47a8-43c2-864c-62fe1dfc298b/revoke-keys/0.log"
Nov 03 13:49:20.460225 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:20.460146    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7a7963b9-ss5rq_5c4b859f-8b05-4cf2-b55f-a3c12987bbda/revoke-keys/0.log"
Nov 03 13:49:20.683936 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:20.683899    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7f01da68-5vwqd_b9c1d268-5e97-432f-b63f-ad41d3d2f4d6/revoke-keys/0.log"
Nov 03 13:49:20.898851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:20.898823    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9b489d3c-4tcqj_a199a9ac-d070-4ca7-a03f-31e45513d2c0/revoke-keys/0.log"
Nov 03 13:49:21.116219 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:21.116192    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ac5024ee-tpk5c_0fda34a8-72d9-4b16-9551-4d6b24c31e80/revoke-keys/0.log"
Nov 03 13:49:21.330584 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:21.330556    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b691b413-8f6n5_5dcec510-4066-463a-8700-b2a961d386d9/revoke-keys/0.log"
Nov 03 13:49:21.551369 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:21.551337    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b99d0fb8-wsgf4_7dd149df-c2ec-4df3-bc61-2e22d1e97b70/revoke-keys/0.log"
Nov 03 13:49:21.778073 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:21.778041    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cbd40680-7wqxs_c339f1d5-217b-4c3e-a4bc-91ea7d91df63/revoke-keys/0.log"
Nov 03 13:49:21.990903 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:21.990875    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d61ce2c5-ptshp_1fabee64-f57b-40c7-acc7-1dfdee1fcb47/revoke-keys/0.log"
Nov 03 13:49:24.967072 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:24.967045    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-7749m_b5f43aa3-c9b8-4f13-bfa7-d875cb364992/kuadrant-console-plugin/0.log"
Nov 03 13:49:27.490540 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:27.490472    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758d88fbcf-s7zpk_046066bc-e7e0-4fa0-8f15-f806f818a2d2/kube-auth-proxy/0.log"
Nov 03 13:49:28.395083 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:28.395050    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hsknc_d962e47d-d4cc-4fa6-b010-1ebf687b3c9e/istio-proxy/0.log"
Nov 03 13:49:29.606268 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:29.606239    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xfwjk_93052df7-c1d3-42d5-8d77-e96e0fd78881/istio-proxy/0.log"
Nov 03 13:49:29.908360 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:29.908279    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-zg2zd_45e0aa94-18d0-4583-8748-75f6100b6272/istio-proxy/0.log"
Nov 03 13:49:30.346940 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:30.346907    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7c679d9f46-jk9ww_94894a65-6ffe-4944-9f4a-04c3c2e10489/payload-pre-processing/0.log"
Nov 03 13:49:31.860916 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:31.860877    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zdd7c_bc4799b4-b7ce-4a2b-bc17-8b54823b2e9c/main/0.log"
Nov 03 13:49:32.555583 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:32.555548    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kpcdf_5b0a55f3-2561-489e-aa3e-ebeb24ca66b4/main/0.log"
Nov 03 13:49:33.196559 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:33.196531    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-mtk74_bc06504b-9351-4e76-971b-31668db4bb5c/main/0.log"
Nov 03 13:49:33.417778 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:33.417723    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lzndt_9e31e600-1afa-400e-8acb-5c32a3c2179d/main/0.log"
Nov 03 13:49:37.650802 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:49:37.650767    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:49:41.058522 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:41.058490    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"]
Nov 03 13:49:41.104051 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:41.104013    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b691b413-8f6n5"]
Nov 03 13:49:42.572761 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:42.572718    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcec510-4066-463a-8700-b2a961d386d9" path="/var/lib/kubelet/pods/5dcec510-4066-463a-8700-b2a961d386d9/volumes"
Nov 03 13:49:43.758695 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.758645    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sv4hs/must-gather-wdsbj"]
Nov 03 13:49:43.759300 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.759278    4546 cpu_manager.go:414] "containerMap: removing container" podUID="1fabee64-f57b-40c7-acc7-1dfdee1fcb47" containerName="revoke-keys"
Nov 03 13:49:43.759348 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.759304    4546 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fabee64-f57b-40c7-acc7-1dfdee1fcb47" containerName="revoke-keys"
Nov 03 13:49:43.759457 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.759445    4546 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fabee64-f57b-40c7-acc7-1dfdee1fcb47" containerName="revoke-keys"
Nov 03 13:49:43.763631 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.763600    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7fst8/must-gather-xvj9c"]
Nov 03 13:49:43.763847 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.763736    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:43.767547 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.767520    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.813386 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.813345    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sv4hs\"/\"openshift-service-ca.crt\""
Nov 03 13:49:43.813585 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.813425    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sv4hs\"/\"kube-root-ca.crt\""
Nov 03 13:49:43.831030 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.830993    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhsm\" (UniqueName: \"kubernetes.io/projected/2213d104-c2ec-48f5-98f9-a86e4770f231-kube-api-access-dfhsm\") pod \"must-gather-wdsbj\" (UID: \"2213d104-c2ec-48f5-98f9-a86e4770f231\") " pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:43.831103 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.831041    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6a919c8-b33d-4340-be8a-7b51d955fc2d-must-gather-output\") pod \"must-gather-xvj9c\" (UID: \"e6a919c8-b33d-4340-be8a-7b51d955fc2d\") " pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.831157 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.831130    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4wf\" (UniqueName: \"kubernetes.io/projected/e6a919c8-b33d-4340-be8a-7b51d955fc2d-kube-api-access-nb4wf\") pod \"must-gather-xvj9c\" (UID: \"e6a919c8-b33d-4340-be8a-7b51d955fc2d\") " pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.831249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.831234    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2213d104-c2ec-48f5-98f9-a86e4770f231-must-gather-output\") pod \"must-gather-wdsbj\" (UID: \"2213d104-c2ec-48f5-98f9-a86e4770f231\") " pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:43.859851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.859816    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sv4hs\"/\"default-dockercfg-pbpvq\""
Nov 03 13:49:43.859851 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.859816    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fst8\"/\"openshift-service-ca.crt\""
Nov 03 13:49:43.860209 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.859948    4546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fst8\"/\"kube-root-ca.crt\""
Nov 03 13:49:43.860280 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.860240    4546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7fst8\"/\"default-dockercfg-mjtw8\""
Nov 03 13:49:43.862698 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.862674    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sv4hs/must-gather-wdsbj"]
Nov 03 13:49:43.865000 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.864979    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/must-gather-xvj9c"]
Nov 03 13:49:43.932182 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.932142    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4wf\" (UniqueName: \"kubernetes.io/projected/e6a919c8-b33d-4340-be8a-7b51d955fc2d-kube-api-access-nb4wf\") pod \"must-gather-xvj9c\" (UID: \"e6a919c8-b33d-4340-be8a-7b51d955fc2d\") " pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.932415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.932206    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2213d104-c2ec-48f5-98f9-a86e4770f231-must-gather-output\") pod \"must-gather-wdsbj\" (UID: \"2213d104-c2ec-48f5-98f9-a86e4770f231\") " pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:43.932415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.932262    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhsm\" (UniqueName: \"kubernetes.io/projected/2213d104-c2ec-48f5-98f9-a86e4770f231-kube-api-access-dfhsm\") pod \"must-gather-wdsbj\" (UID: \"2213d104-c2ec-48f5-98f9-a86e4770f231\") " pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:43.932415 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.932289    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6a919c8-b33d-4340-be8a-7b51d955fc2d-must-gather-output\") pod \"must-gather-xvj9c\" (UID: \"e6a919c8-b33d-4340-be8a-7b51d955fc2d\") " pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.932596 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.932575    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2213d104-c2ec-48f5-98f9-a86e4770f231-must-gather-output\") pod \"must-gather-wdsbj\" (UID: \"2213d104-c2ec-48f5-98f9-a86e4770f231\") " pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:43.932669 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.932647    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6a919c8-b33d-4340-be8a-7b51d955fc2d-must-gather-output\") pod \"must-gather-xvj9c\" (UID: \"e6a919c8-b33d-4340-be8a-7b51d955fc2d\") " pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.988395 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.988365    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4wf\" (UniqueName: \"kubernetes.io/projected/e6a919c8-b33d-4340-be8a-7b51d955fc2d-kube-api-access-nb4wf\") pod \"must-gather-xvj9c\" (UID: \"e6a919c8-b33d-4340-be8a-7b51d955fc2d\") " pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:43.988601 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:43.988373    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhsm\" (UniqueName: \"kubernetes.io/projected/2213d104-c2ec-48f5-98f9-a86e4770f231-kube-api-access-dfhsm\") pod \"must-gather-wdsbj\" (UID: \"2213d104-c2ec-48f5-98f9-a86e4770f231\") " pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:44.080779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:44.080680    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sv4hs/must-gather-wdsbj"
Nov 03 13:49:44.085165 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:44.085145    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/must-gather-xvj9c"
Nov 03 13:49:44.470959 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:44.470936    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sv4hs/must-gather-wdsbj"]
Nov 03 13:49:44.472849 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:49:44.472813    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2213d104_c2ec_48f5_98f9_a86e4770f231.slice/crio-c615bae28ed6d23192be3f24dd2b8e315c35dbd93cde2f21a3d9652add55267c WatchSource:0}: Error finding container c615bae28ed6d23192be3f24dd2b8e315c35dbd93cde2f21a3d9652add55267c: Status 404 returned error can't find the container with id c615bae28ed6d23192be3f24dd2b8e315c35dbd93cde2f21a3d9652add55267c
Nov 03 13:49:44.474766 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:44.474734    4546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:49:44.518784 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:44.518737    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/must-gather-xvj9c"]
Nov 03 13:49:44.521206 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:49:44.521169    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a919c8_b33d_4340_be8a_7b51d955fc2d.slice/crio-3421d85fd7c8b0694827e8fa96293fadbe6bd10e6c62b82e376d23a0a47407e8 WatchSource:0}: Error finding container 3421d85fd7c8b0694827e8fa96293fadbe6bd10e6c62b82e376d23a0a47407e8: Status 404 returned error can't find the container with id 3421d85fd7c8b0694827e8fa96293fadbe6bd10e6c62b82e376d23a0a47407e8
Nov 03 13:49:45.153734 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:45.153690    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sv4hs/must-gather-wdsbj" event={"ID":"2213d104-c2ec-48f5-98f9-a86e4770f231","Type":"ContainerStarted","Data":"c615bae28ed6d23192be3f24dd2b8e315c35dbd93cde2f21a3d9652add55267c"}
Nov 03 13:49:45.155089 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:45.155068    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/must-gather-xvj9c" event={"ID":"e6a919c8-b33d-4340-be8a-7b51d955fc2d","Type":"ContainerStarted","Data":"3421d85fd7c8b0694827e8fa96293fadbe6bd10e6c62b82e376d23a0a47407e8"}
Nov 03 13:49:46.163264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:46.163175    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/must-gather-xvj9c" event={"ID":"e6a919c8-b33d-4340-be8a-7b51d955fc2d","Type":"ContainerStarted","Data":"5178ab2543978bde8be9acfd7c7511dc27fe62d5ce198243d213107c75720c8e"}
Nov 03 13:49:46.163264 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:46.163228    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/must-gather-xvj9c" event={"ID":"e6a919c8-b33d-4340-be8a-7b51d955fc2d","Type":"ContainerStarted","Data":"e3ce36aafdfabc0bc5dd927f24d5019f09c0beb88c4927ce9128e57389ce7631"}
Nov 03 13:49:46.270483 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:46.270427    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7fst8/must-gather-xvj9c" podStartSLOduration=1.931736685 podStartE2EDuration="3.270410952s" totalImagesPullingTime="1.338674267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:46.268528411 +0000 UTC m=+7127.840112335" watchObservedRunningTime="2026-11-03 13:49:46.270410952 +0000 UTC m=+7127.841994866"
Nov 03 13:49:52.062386 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:52.062338    4546 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"]
Nov 03 13:49:52.107705 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:52.107675    4546 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48061e9f-xs8pq"]
Nov 03 13:49:52.574796 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:52.574713    4546 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01463f96-983a-4f86-b409-c91e1e09e51e" path="/var/lib/kubelet/pods/01463f96-983a-4f86-b409-c91e1e09e51e/volumes"
Nov 03 13:49:53.696773 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:53.696667    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lpbww_606502b8-aa74-4661-94fe-43d62bfb8d40/global-pull-secret-syncer/0.log"
Nov 03 13:49:57.053174 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:57.053106    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wh554_2ead3a7f-804d-48a7-9dae-ec5df206cbbe/konnectivity-agent/0.log"
Nov 03 13:49:57.883085 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:49:57.883048    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-245.us-west-2.compute.internal_5cd06e439bdd68be11ac5140e2509e6a/haproxy/0.log"
Nov 03 13:50:00.320249 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:00.320205    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sv4hs/must-gather-wdsbj" event={"ID":"2213d104-c2ec-48f5-98f9-a86e4770f231","Type":"ContainerStarted","Data":"164efc627e6b8ec66382753936365ca96872f547650845247a675c31ef7faf8d"}
Nov 03 13:50:00.320827 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:00.320263    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sv4hs/must-gather-wdsbj" event={"ID":"2213d104-c2ec-48f5-98f9-a86e4770f231","Type":"ContainerStarted","Data":"9c2df9a3a206bcc06bbdab3110b4e8a921277084bef6e42f5466a24bd441be1d"}
Nov 03 13:50:03.709017 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:03.708977    4546 scope.go:122] "RemoveContainer" containerID="81457cc8ad5464e4d7db59289369d5aea9cfa291e38ec9fcf3964eac358e642f"
Nov 03 13:50:03.719562 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:03.719539    4546 scope.go:122] "RemoveContainer" containerID="d8e7a6603c7ce1a811c1118a2c293976f9e119bd5aebb233f7ddf3501a5a2f28"
Nov 03 13:50:03.730937 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:03.730914    4546 scope.go:122] "RemoveContainer" containerID="d035d99cdb444a084d24459b9eb19bf8c15f4f43c0add9f5d5f2f4a501be387e"
Nov 03 13:50:04.357075 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.357031    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cd79d8544-f6nxm_abc3e369-07e6-4332-a68a-58c77461df97/manager/0.log"
Nov 03 13:50:04.719239 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.719185    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-sv4hs/must-gather-wdsbj" podStartSLOduration=7.00437842 podStartE2EDuration="21.719171821s" totalImagesPullingTime="14.714793401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:00.433199722 +0000 UTC m=+7142.004783636" watchObservedRunningTime="2026-11-03 13:50:04.719171821 +0000 UTC m=+7146.290755733"
Nov 03 13:50:04.721779 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.721742    4546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"]
Nov 03 13:50:04.729535 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.729511    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.813683 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.813644    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"]
Nov 03 13:50:04.849707 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.849671    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqx6s\" (UniqueName: \"kubernetes.io/projected/59562cf5-dd6d-4dae-a2f8-b3df5252698b-kube-api-access-zqx6s\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.849912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.849735    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-lib-modules\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.849912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.849857    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-podres\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.849912 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.849894    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-sys\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.850043 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.849981    4546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-proc\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951181 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951134    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-lib-modules\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951228    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-podres\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951268    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-sys\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951294    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-lib-modules\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951313    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-proc\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951385 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951372    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-proc\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951431    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-sys\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.951628 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951502    4546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqx6s\" (UniqueName: \"kubernetes.io/projected/59562cf5-dd6d-4dae-a2f8-b3df5252698b-kube-api-access-zqx6s\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:04.952004 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:04.951984    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59562cf5-dd6d-4dae-a2f8-b3df5252698b-podres\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:05.005127 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:05.005050    4546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqx6s\" (UniqueName: \"kubernetes.io/projected/59562cf5-dd6d-4dae-a2f8-b3df5252698b-kube-api-access-zqx6s\") pod \"perf-node-gather-daemonset-xrbf9\" (UID: \"59562cf5-dd6d-4dae-a2f8-b3df5252698b\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:05.042416 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:05.042375    4546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:05.283320 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:05.283238    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-7749m_b5f43aa3-c9b8-4f13-bfa7-d875cb364992/kuadrant-console-plugin/0.log"
Nov 03 13:50:05.434067 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:05.434031    4546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"]
Nov 03 13:50:05.438840 ip-10-0-136-245 kubenswrapper[4546]: W1103 13:50:05.438805    4546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod59562cf5_dd6d_4dae_a2f8_b3df5252698b.slice/crio-8d5be4a9fb28b0cefd236848f88278abe9cc56bfbe864e4f3d9cd9eaeba9de14 WatchSource:0}: Error finding container 8d5be4a9fb28b0cefd236848f88278abe9cc56bfbe864e4f3d9cd9eaeba9de14: Status 404 returned error can't find the container with id 8d5be4a9fb28b0cefd236848f88278abe9cc56bfbe864e4f3d9cd9eaeba9de14
Nov 03 13:50:05.690740 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:05.690705    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c58c9bfd-64vqd_e794e88c-e1c1-4534-9bb5-bbc0a8c40773/manager/0.log"
Nov 03 13:50:06.374330 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:06.374284    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9" event={"ID":"59562cf5-dd6d-4dae-a2f8-b3df5252698b","Type":"ContainerStarted","Data":"71eedceb04ef75244c78da1403ff7d00466ee360d3b24ae3f0ba7c49d807427e"}
Nov 03 13:50:06.374975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:06.374366    4546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9" event={"ID":"59562cf5-dd6d-4dae-a2f8-b3df5252698b","Type":"ContainerStarted","Data":"8d5be4a9fb28b0cefd236848f88278abe9cc56bfbe864e4f3d9cd9eaeba9de14"}
Nov 03 13:50:06.374975 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:06.374516    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:06.486606 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:06.486507    4546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9" podStartSLOduration=2.48649044 podStartE2EDuration="2.48649044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:06.483939974 +0000 UTC m=+7148.055523884" watchObservedRunningTime="2026-11-03 13:50:06.48649044 +0000 UTC m=+7148.058074352"
Nov 03 13:50:07.651996 ip-10-0-136-245 kubenswrapper[4546]: E1103 13:50:07.651909    4546 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c659" containerName="extractor"
Nov 03 13:50:09.979767 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:09.979726    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cd79d8544-f6nxm_abc3e369-07e6-4332-a68a-58c77461df97/manager/0.log"
Nov 03 13:50:11.438694 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:11.438661    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c58c9bfd-64vqd_e794e88c-e1c1-4534-9bb5-bbc0a8c40773/manager/0.log"
Nov 03 13:50:12.399563 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:12.399530    4546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-xrbf9"
Nov 03 13:50:14.165445 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:14.165407    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-b424q_e81db497-fb6d-4f0f-9568-6eb450d61851/monitoring-plugin/0.log"
Nov 03 13:50:15.799886 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:15.799856    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpbk9_17f1df83-bba7-4eae-8a25-bfb807a364c5/node-exporter/0.log"
Nov 03 13:50:16.206277 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:16.206191    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpbk9_17f1df83-bba7-4eae-8a25-bfb807a364c5/kube-rbac-proxy/0.log"
Nov 03 13:50:16.221216 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:16.221178    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cd79d8544-f6nxm_abc3e369-07e6-4332-a68a-58c77461df97/manager/0.log"
Nov 03 13:50:16.694661 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:16.694626    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpbk9_17f1df83-bba7-4eae-8a25-bfb807a364c5/init-textfile/0.log"
Nov 03 13:50:17.571146 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:17.571104    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c58c9bfd-64vqd_e794e88c-e1c1-4534-9bb5-bbc0a8c40773/manager/0.log"
Nov 03 13:50:19.557114 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:19.557081    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/prometheus/0.log"
Nov 03 13:50:19.963473 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:19.963394    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/config-reloader/0.log"
Nov 03 13:50:20.368309 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:20.368281    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/thanos-sidecar/0.log"
Nov 03 13:50:20.775383 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:20.775356    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/kube-rbac-proxy-web/0.log"
Nov 03 13:50:21.187253 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:21.187172    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/kube-rbac-proxy/0.log"
Nov 03 13:50:21.595166 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:21.595136    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/kube-rbac-proxy-thanos/0.log"
Nov 03 13:50:22.002282 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:22.002250    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0161452-0240-42ac-8238-f85ec8057284/init-config-reloader/0.log"
Nov 03 13:50:23.222582 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:23.222552    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lztc9_ac1369cc-8144-46c9-a267-c4586d82f99e/prometheus-operator-admission-webhook/0.log"
Nov 03 13:50:25.152142 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:25.152109    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758d88fbcf-s7zpk_046066bc-e7e0-4fa0-8f15-f806f818a2d2/kube-auth-proxy/0.log"
Nov 03 13:50:26.025001 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:26.024973    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hsknc_d962e47d-d4cc-4fa6-b010-1ebf687b3c9e/istio-proxy/0.log"
Nov 03 13:50:27.110580 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:27.110548    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xfwjk_93052df7-c1d3-42d5-8d77-e96e0fd78881/istio-proxy/0.log"
Nov 03 13:50:27.334609 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:27.334570    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-zg2zd_45e0aa94-18d0-4583-8748-75f6100b6272/istio-proxy/0.log"
Nov 03 13:50:27.764393 ip-10-0-136-245 kubenswrapper[4546]: I1103 13:50:27.764361    4546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7c679d9f46-jk9ww_94894a65-6ffe-4944-9f4a-04c3c2e10489/payload-pre-processing/0.log"